Add a flag to set up test environment only once per test class; introduce setUpBeforeClass() and tearDownAfterClass() [PHPUnit]

Created on 20 January 2012, almost 13 years ago
Updated 21 March 2024, 9 months ago

Repeat: How many tests will actually fail if setUp() is only executed once for a test case?

This patch cuts down the total time for the full Drupal core test suite to 9 minutes. (from currently 20+ minutes)

API changes

  1. The current setUp() and tearDown() methods are replaced with setUpBeforeClass() and tearDownAfterClass():
    -  function setUp() {
    -    parent::setUp(array('taxonomy', 'node'));
    +  function setUpBeforeClass() {
    +    parent::setUpBeforeClass(array('taxonomy', 'node'));
    
  2. setUpBeforeClass() and tearDownAfterClass() are only invoked once for each test class.
  3. setUp() and tearDown() are however retained, and invoked before and after each test method in a class.
  4. All test methods in a class share the same test environment.
  5. Test methods are executed in the order they are defined.
  6. Test methods need to ensure that they are either executed in the right order, or revert any special conditions that they have changed, so following test methods being executed afterwards still work.

    This especially applies to configuration or context that is set up once for test methods and is shared between them (so called "fixtures").

    Example for stuff that breaks:

    class WhateverTest extends WebTestBase {
      function setUpBeforeClass() {
        parent::setUpBeforeClass();
        $this->admin_user = $this->drupalCreateUser();
        // If any test method logs the user out, following tests will break.
        // Move this into setUp(), so admin_user is logged in for each test method.
        $this->drupalLogin($this->admin_user);
    
        $this->node = $this->drupalCreateNode();
      }
    
      function testMaintenanceAccess() {
        // Maintenance mode will be enabled for all following test methods, which
        // obviously makes most tests fail.
        // Reset or delete the variable to its original/default value at the end of this test.
        variable_set('maintenance_mode', 1);
      }
    
      function testDelete() {
        // This deletes the shared node and thus breaks testResave().
        // Move/relocate this test method after testResave() - or alternatively,
        // take the appropriate measures to restore the expected data at the end of
        // this test (which either means to update $this->node with a new one, or,
        // in case the test requires it, even with the identical ID $this->node->nid).
        node_delete($this->node->nid);
        $this->assertFalse(node_load($this->node->nid), FALSE);
      }
    
      function testResave() {
        $resaved_node = node_load($this->node->nid);
        node_save($resaved_node);
        $this->assertIdentical($this->node->nid, $resaved_node->nid);
      }
    }
    

Help to get this done

@sun will not be able to champion all of the required test changes on his own. But there is a sandbox, so you can help! :)

How to help:

  1. Ask for git write access to the Platform sandbox, if you haven't already. IRC is fastest; but a comment here works, too.
  2. Setup the Platform sandbox for your local D8 git repo/clone.
  3. Checkout @sun's branch into a branch that's specific to you:
    git checkout -b test-once-1411074-[username] platform/test-once-1411074-sun
    git push -u platform test-once-1411074-[username]
    

    This means you're using @sun's branch as the tip to work off from. @sun will review your changes and merge them into the main branch, if appropriate.

  4. To create patches for the testbot, diff against the -base branch:
    git diff platform/test-once-base test-once-1411074-[username]
    
  5. To merge new changes from a test-once branch from another user:
    git merge --no-ff platform/test-once-1411074-[otheruser]
    

    The --no-ff option is key here.

  6. Operations you are not allowed to do:
    • git push platform without specifying a branch name: This would push all of your local branches into the platform sandbox.
    • git push platform [8.x:]test-once-base: Only @sun updates the base branch when needed.
    • git push platform test-once-1411074-sun: You only push to the branch that has your username.
    • git rebase: Rewrites history and makes your branch incompatible / unmergeable.
    • git pull 8.x: Merges in all latest 8.x code into your test-once branch, but all work is based on the -base branch.
    • git merge 8.x: Ditto.
📌 Task
Status

Needs work

Version

11.0 🔥

Component
PHPUnit 

Last updated 3 days ago

Created by

🇩🇪Germany sun Karlsruhe

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇬🇧United Kingdom joachim

    This should be resurrected - but should be split into separate issues for Kernel / Browser / JS tests.

  • 🇺🇸United States mile23 Seattle, WA
  • 🇷🇺Russia Chi

    It feels if we had fixed this issue in 2012 the Drupal Association could save tens of thousands of dollars on CI environments.

  • 🇬🇧United Kingdom joachim

    I discussed this with @Alex Pott at DrupalCon Lille a few months ago.

    His comment IIRC was along the lines of 'you can usually just comment parts out of the code'.

    Here's an example where AFAICT, you can't. I want to test all the possible combinations of 4 factors. So I'm using a cross product setup. To do so with a @dataProvider would mean a kernel test setup for each combination, which is needlessly expensive.

    So I'm doing the cross product manually in the test method like this:

        foreach ([FALSE, TRUE] as $permission_access) {
          foreach ([FALSE, TRUE] as $operand_access) {
            foreach ([FALSE, TRUE] as $operability) {
              foreach ([FALSE, TRUE] as $reachable) {
                $this->state->set('test_mocked_control:permission_access', $permission_access ? AccessResult::allowed() : AccessResult::forbidden());
                $this->state->set('test_mocked_control:operand_access', $operand_access ? AccessResult::allowed() : AccessResult::forbidden());
                $this->state->set('test_mocked_control:operability', $operability);
                $this->state->set('test_mocked_control:next_state', $reachable ? 'cake' : NULL);
    
                // Generate the links.
                $links = $action_link->getStateActionPlugin()->buildLinkArray($action_link, $user_no_access);
    
                // No access always means no links.
                if (!$permission_access || !$operand_access()) {
                  $this->assertEmpty($links, implode(':', [$permission_access, $operand_access, $operability, $reachable]));
                }
    

    But this has a number of drawbacks:

    - I have to ensure each assertion has a custom message otherwise I will have no idea which set of values caused a failure
    - furthermore, I have to use cross product values which are easily stringable, so the message can be easily made without complex code. This harms DX, as it would have been easier to read if the values for the access factors were AccessResult objects -- the checks further down would be easier to read than the boolean check for instance
    - I can't run just a single combination for debugging. Or well, I can if I rewrite the cross product code like this so that values can be commented out, but it still requires a lot of faffing about to isolate a single case:

        $permission_access_values = [
          FALSE,
          TRUE,
        ];
        $operand_access_values = [
          FALSE,
          TRUE,
        ];
        $operability_values = [
          FALSE,
          TRUE,
        ];
        $reachable_values = [
          FALSE,
          TRUE,
        ];
    
        foreach ($permission_access_values as $permission_access) {
          foreach ($operand_access_values as $operand_access) {
            foreach ($operability_values as $operability) {
              foreach ($reachable_values as $reachable) {
    

    All of this is reinventing the wheel which PHPUnit provides with the data provider system.

  • 🇨🇦Canada mgifford Ottawa, Ontario

    How often are we running these tests? Seems like a good investment of time to cut the execution time in half, even if it's a matter of allowing us to know earlier that the patch works.

Production build 0.71.5 2024