Don't bother with file copies during build, keep file paths on the build and deploy from the original file

Created on 27 March 2022, over 2 years ago
Updated 19 April 2023, about 1 year ago

Problem/Motivation

When a path is processed by the tome generator, any files (JS, CSS, images) attached are copies from their original location into the staging area for the preview site.
For a non trivial node (e.g. a page with a complex react application) this could result in one path requiring 100 files to be copied.
On a distributed file system like EFS, copying 100 files as well as any other processing for that path may not complete in the allowed timeout for a single request

Steps to reproduce

Configure local host to have a 60s timeout
Edit tome's static generator class's copyPath method to add a 300ms sleep after each file copy to emulate latency seen in a distributed file system
Attempt to generate a preview for a piece of content with a complex FE component/ React app
Experience timeout

Proposed resolution

We already sub-class tome's static generator and have an implementation of copyPaths that tracks copied files.
Instead of calling the parent in this method, push the file paths to the build.
Have the deploy process deploy them from the original file instead of a copied version

Remaining tasks

User interface changes

API changes

Data model changes

πŸ› Bug report
Status

Fixed

Version

1.0

Component

Code

Created by

πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    First a re-roll

  • The last submitted patch, 8: 3271913-8.patch, failed testing. View results β†’
    - codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    New approach, no interdiff as its a do-over

  • The last submitted patch, 11: 3271913-2.patch, failed testing. View results β†’
    - codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Allow for subdirectory on test-bot

  • The last submitted patch, 15: 3271913-10.patch, failed testing. View results β†’
    - codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Take two

  • πŸ‡¦πŸ‡ΊAustralia acbramley
    --- a/modules/preview_site_s3/src/Plugin/PreviewSite/Deploy/S3.php
    +++ b/modules/preview_site_s3/src/Plugin/PreviewSite/Deploy/S3.php
    
    @@ -269,4 +269,32 @@ class S3 extends DeployPluginBase implements ContainerFactoryPluginInterface {
    +  public function deployFilePath(PreviewSiteBuildInterface $build, string $path): void {
    

    This is almost identical to deployArtifact

    Can we reafctor that so it calls this with FileHelper::getFilePathWithoutSchema($file, $build) as the path?

    Otherwise looking pretty good, great idea!

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Can we reafctor that so it calls this with FileHelper::getFilePathWithoutSchema($file, $build) as the path?

    I considered it, but was on the fence, but having another vote for doing it helps sway me towards it. Will do!

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Made that change and fixed some PHPCS while I was at it

  • πŸ‡¦πŸ‡ΊAustralia mortim07

    I mentioned this to @larowlan, but this has also solved OOM issues with the preview site build form - during validation it was validating the entity reference for all artifacts.

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Cutting a new release with this

  • Status changed to Fixed over 1 year ago
    • larowlan β†’ committed b292dcc6 on 1.x
      Issue #3271913 by larowlan, dpi, acbramley, mortim07: Don't bother with...
  • Status changed to Active over 1 year ago
  • πŸ‡΅πŸ‡±Poland kamil_lw

    Hello,

    I'm reopening this issue, because I'm experiencing a PHP Fatal Error.

    Here's a part of the stack trace:

     Error: Call to undefined method Drupal\preview_site\Generate\TomeStaticExtension::getJavascriptModules() in /opt/drupal/web/modules/contrib/preview_site/src/Generate/TomeStaticExtension.php:185 Stack trace: 
    #0 /opt/drupal/web/modules/contrib/preview_site/src/Plugin/PreviewSite/Generate/TomeGenerator.php(207): Drupal\preview_site\Generate\TomeStaticExtension->exportPaths(Array) 
    #1 /opt/drupal/web/modules/contrib/preview_site/src/Plugin/PreviewSite/Generate/TomeGenerator.php(129): Drupal\preview_site\Plugin\PreviewSite\Generate\TomeGenerator->generateBuildForPath(Object(Drupal\preview_site\Entity\PreviewSiteBuild), '_entity:node:en...', 'http://default/...', Object(Drupal\Core\Queue\DatabaseQueue)) 
    #2 /opt/drupal/web/modules/contrib/preview_site/src/Plugin/QueueWorker/Generate.php(115): Drupal\preview_site\Plugin\PreviewSite\Generate\TomeGenerator->generateBuildForItem(Object(Drupal\preview_site\Entity\PreviewSiteBuild), Object(Drupal\dynamic_entity_reference\Plugin\Field\FieldType\DynamicEntityReferenceItem), 'http://default/...', Object(Drupal\Core\Queue\DatabaseQueue)) 
    #3 /opt/drupal/web/modules/contrib/preview_site/src/PreviewSiteBuilder.php(180): Drupal\preview_site\Plugin\QueueWorker\Generate->processItem(0) 
    #4 /opt/drupal/web/modules/contrib/preview_site/src/PreviewSiteBuilder.php(120): Drupal\preview_site\PreviewSiteBuilder->processQueueItem('preview_site_ge...') 
    #5 /opt/drupal/web/modules/contrib/preview_site/src/PreviewSiteBuilder.php(242): Drupal\preview_site\PreviewSiteBuilder->processSiteGeneration(Object(Drupal\preview_site\Entity\PreviewSiteBuild)) 
    #6 /opt/drupal/packages/languagewire_tmgmt_connector/src/Adapter/PreviewSite/PreviewSiteBuilder.php(66): Drupal\preview_site\PreviewSiteBuilder::operationProcessGenerate(303, Array)
    (...)
    

    I've taken a look at the preview_site module source code, and I indeed can find the "getJavascriptModules()" method definition nowhere in there.

    This issue was introduced in the release 1.1.5 of the module.

    PS. This is my first time adding a comment in Drupal's issue tracker. Please forgive me if I should have opened a new issue instead.

  • πŸ‡¦πŸ‡ΊAustralia acbramley

    You're right, that function is provided by https://www.drupal.org/project/tome/issues/3230078 ✨ Add support for Javascript modules Fixed

    I'm surprised tests are passing...

  • πŸ‡¦πŸ‡ΊAustralia mortim07

    @acbramley There isn't test coverage for it. Do we remove the patch for now and roll a new release until the dependency is merged in?

  • Status changed to Needs work over 1 year ago
  • πŸ‡¦πŸ‡ΊAustralia acbramley

    I've reverted the commit and published 1.1.6

  • πŸ‡¦πŸ‡ΊAustralia mortim07

    Great @acbramley!

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Let's try and push that one forward then

  • Status changed to Needs review about 1 year ago
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    The tome issue is fixed/released so here's a new patch with a minimum requirement on that version of tome

  • Status changed to RTBC about 1 year ago
  • πŸ‡¦πŸ‡ΊAustralia acbramley

    Awesome! RTBC assuming green :)

    • larowlan β†’ committed 14010b07 on 1.x
      Issue #3271913 by larowlan, dpi, acbramley, mortim07, kamil_lw: Don't...
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Ran the tests locally, as something not right with drupalci, the folder is missing πŸ€”

    phpunit -c app/core app/modules/preview_site --stop-on-failure
    ⏳️ Bootstrapped tests in 0 seconds.
    🐘 PHP Version 8.1.16.
    πŸ’§ Drupal Version 9.5.8-dev.
    πŸ—„οΈ  Database engine mysql.
    PHPUnit 9.6.6 by Sebastian Bergmann and contributors.
    
    Testing /data/app/modules/preview_site
    ................................................................. 65 / 66 ( 98%)
    .                                                                 66 / 66 (100%)
    
    Time: 03:17.707, Memory: 18.00 MB
    
    OK (66 tests, 1615 assertions)
    
    
  • Status changed to Fixed about 1 year ago
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.69.0 2024