Add assertions to OpenTelemetryFrontPagePerformanceTest::testFrontPageColdCache()

Created on 24 September 2024, 3 months ago

Problem/Motivation

Note: this issue was initially created for a performance testing workshop at DrupalCon Barcelona.

OpenTelemetryFrontPagePerformanceTest::testFrontPageColdCache() doesn't have assertions for cache gets/sets, database queries, and Javascript and CSS requests.

Steps to reproduce

Proposed resolution

Using OpenTelemetryFrontPagePerformanceTest::testFrontPageHotCache(), AssetAggregationAcrossPagesTest::testFrontAndRecipesPages(), and OpenTelemetryAuthenticatedPerformanceTest::testFrontPageAuthenticatedWarmCache() as examples, add assertions to these test methods for database queries, cache operations, and CSS/JS requests - the tests should continue to pass with the assertions that are added.

Remaining tasks

User interface changes

N/A

Introduced terminology

N/A

API changes

N/A

Data model changes

N/A

Release notes snippet

N/A

πŸ“Œ Task
Status

Active

Version

11.0 πŸ”₯

Component

phpunit

Created by

πŸ‡¬πŸ‡§United Kingdom catch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @catch
  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I plan to work on this issue here at DrupalCon Barcelona.

  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I created a draft MR. I commented out the assertions that seem to fail, as well as a var_dump() line.

  • Pipeline finished with Failed
    3 months ago
    Total: 155s
    #291520
  • πŸ‡§πŸ‡¬Bulgaria pfrenssen Sofia

    The commented out code trips up PHP_CodeSniffer, and this causes the pipeline to exit early. Let's do a test run without the commented out lines so we can see the PHPUnit test results.

  • Pipeline finished with Success
    3 months ago
    Total: 403s
    #292456
  • Pipeline finished with Failed
    3 months ago
    Total: 405s
    #292480
  • Pipeline finished with Failed
    3 months ago
    Total: 651s
    #292503
  • πŸ‡§πŸ‡¬Bulgaria pfrenssen Sofia

    I added back some of the metric that were commented out. In my local testing it turns out to be rather unstable, and some of the value ranges are wider than I like, for example:

        $this->assertCountBetween(600, 750, $performance_data->getCacheGetCount());
    

    These cache hits are not predictable since the browser will request CSS / JS assets and generate image styles. We have no control over when these additional requests are happening within our test window.

    Even with these wide ranges I think we will see random failures happening. To minimize the chances of this happening we should set such wide ranges that the tests will become useless.

    I think the right solution for these "cold cache" tests is to take the browser cache out of scope, and focus only on the Drupal cache:

        // Make sure all aggregated assets and image styles are generated and cached in the browser.
        $this->drupalGet('<front>');
        $this->drupalGet('<front>')
        // Clear caches to ensure a cold Drupal cache.
        $this->rebuildAll();
    

    This will make the results more reliable and actually usable.

    There is still value in doing a fully cold (drupal + browser) cache test, but then we should not inspect the cache hits.

  • πŸ‡¬πŸ‡§United Kingdom catch

    @pfrenssen

    I think the right solution for these "cold cache" tests is to take the browser cache out of scope, and focus only on the Drupal cache

    Yes completely agreed with this. We can open an issue to add coverage specifically for asset and image derivative route performance testing, but no need to test both in this one.

  • Pipeline finished with Canceled
    3 months ago
    Total: 127s
    #293253
  • Pipeline finished with Failed
    3 months ago
    Total: 496s
    #293255
  • πŸ‡§πŸ‡¬Bulgaria pfrenssen Sofia

    There is a failure caused by a missing aggregated CSS file:

    1)
        Drupal\Tests\demo_umami\FunctionalJavascript\OpenTelemetryFrontPagePerformanceTest::testFrontPagePerformance
        TypeError: str_replace(): Argument #3 ($subject) must be of type
        array|string, false given
        
        /builds/issue/drupal-3476423/core/tests/Drupal/Tests/PerformanceTestTrait.php:391
        /builds/issue/drupal-3476423/core/tests/Drupal/Tests/PerformanceTestTrait.php:333
        /builds/issue/drupal-3476423/core/tests/Drupal/Tests/PerformanceTestTrait.php:115
        /builds/issue/drupal-3476423/core/profiles/demo_umami/tests/src/FunctionalJavascript/OpenTelemetryFrontPagePerformanceTest.php:114
        /builds/issue/drupal-3476423/core/profiles/demo_umami/tests/src/FunctionalJavascript/OpenTelemetryFrontPagePerformanceTest.php:31
        
        --
        
        1 test triggered 1 PHP warning:
        
        1)
        /builds/issue/drupal-3476423/core/tests/Drupal/Tests/PerformanceTestTrait.php:391
        file_get_contents(sites/simpletest/19116417/files/css/css_So59JtkW7L7nLWoPscWwEf2eVnGINAq6NAsCC81gGnA.css):
        Failed to open stream: No such file or directory
        
        Triggered by:
        
        *
        Drupal\Tests\demo_umami\FunctionalJavascript\OpenTelemetryFrontPagePerformanceTest::testFrontPagePerformance
         
        /builds/issue/drupal-3476423/core/profiles/demo_umami/tests/src/FunctionalJavascript/OpenTelemetryFrontPagePerformanceTest.php:27
    

    This is probably caused by the preceding call to $this->rebuildAll() which does a full drupal cache flush, including purging the aggregated CSS and JS files.

  • Pipeline finished with Success
    3 months ago
    Total: 3340s
    #293345
  • πŸ‡§πŸ‡¬Bulgaria pfrenssen Sofia

    I ran the test locally 4 times in a row and the values were stable. I guess it's OK to check the actual values.

Production build 0.71.5 2024