Run phpunit tests from a single job in parallel

Core made an attempt to try this in its earlier versions here 📌 [experiment] Explore paratest to run our phpunit tests in parallel Needs work

So far, I've only found one potential case that the library doesn't support, which is the use of custom phpunit printers, which we use here
https://git.drupalcode.org/project/gitlab_templates/-/blob/1.0.x/include...

I made a comment on an open issue to see if they'd allow for this option to be overriden: https://github.com/paratestphp/paratest/issues/771

Due to this, if we run the tests using this tool, we won't have any "browser_output" files.

I think we can live with that, that just creates a ton of artifacts that are very rarely needed, I think for core they are already now not accessible.

What could be neat is having some kind of debug mode, that easily allows to run just a single test/group/something with browser output and then not in parallel. I think manual jobs have an ability to pass in parameters?

Oh, that's really good to know then.

having some kind of debug mode, that easily allows to run just a single test/group/something with browser output and then not in parallel

That's what we have right now with this line
vendor/bin/phpunit --no-interaction --printer="\Drupal\Tests\Listeners\HtmlOutputPrinter" --bootstrap $PWD/web/core/tests/bootstrap.php $_WEB_ROOT/modules/custom --log-junit junit.xml $_PHPUNIT_EXTRA

That generates the full output into the "browser_outout" folder and is available as artifact. See this run https://git.drupalcode.org/project/api/-/jobs/54026, where we are using the default phpunit - non parallel -, where we have the browser output available.

The $_PHPUNIT_EXTRA variable would allow us to pass additional options, like groups for example.

I will continue playing with the library and see the possibilities within these templates.

Anybody willing to try this would be great.

You can test it in your module's ".gitlab-ci.yml" like this:

variables:
  _PHPUNIT_CONCURRENT:
    value: "1"

.show-environment-variables: &show-environment-variables
    - |
      echo -e "\e[0Ksection_start:`date +%s`:show_env_vars[collapsed=true]\r\e[0KShow Environment Variables"
      env | sort
      echo -e "\e[0Ksection_end:`date +%s`:show_env_vars\r\e[0K"

.simpletest_db: &simpletest-db
  - |
    [[ $_TARGET_DB_TYPE == "sqlite" ]] && export SIMPLETEST_DB=sqlite://localhost/sites/default/files/db.sqlite
    [[ $_TARGET_DB_TYPE == "mysql" ]] && export SIMPLETEST_DB=mysql://$MYSQL_USER:$MYSQL_PASSWORD@database/$MYSQL_DATABASE
    [[ $_TARGET_DB_TYPE == "pgsql" ]] && export SIMPLETEST_DB=pgsql://$POSTGRES_USER:$POSTGRES_PASSWORD@database/$POSTGRES_DB

.phpunit-base:
  script:
    - *simpletest-db
    - *show-environment-variables
    - mkdir -p $_WEB_ROOT/sites/simpletest/browser_output
    - chmod -R 755 $_WEB_ROOT/sites/simpletest
    - touch $_WEB_ROOT/webserver.log
    # @todo Use Apache instead of PHP web server.
    - cd $_WEB_ROOT && php -S 0.0.0.0:8888 .ht.router.php >> webserver.log 2>&1 &
    # Provide some context on the test run.
    - vendor/bin/drush status
    # Finally, execute tests.
    - |
      [[ $_PHPUNIT_CONCURRENT == "1" ]] && composer require --dev brianium/paratest:~6
    - |
      [[ $_PHPUNIT_CONCURRENT == "1" ]] && vendor/bin/paratest --functional -v --bootstrap $PWD/web/core/tests/bootstrap.php $_WEB_ROOT/modules/custom --log-junit junit.xml $_PHPUNIT_EXTRA
    - |
      [[ $_PHPUNIT_CONCURRENT == "0" ]] && vendor/bin/phpunit --no-interaction --printer="\Drupal\Tests\Listeners\HtmlOutputPrinter" --bootstrap $PWD/web/core/tests/bootstrap.php $_WEB_ROOT/modules/custom --log-junit junit.xml $_PHPUNIT_EXTRA

Then commit and push the changes.

You can also test it locally (https://bjorn.dev/blog/drupal-gitlab-ci-running-contrib-tests-locally/), like this:

gitlab-ci-local --remote-variables git@git.drupal.org:issue/gitlab_templates-3370952=includes/include.drupalci.variables.yml=1.0.x   --variable="_GITLAB_TEMPLATES_REPO=issue/gitlab_templates-3370952"   --variable="_GITLAB_TEMPLATES_REF=1.0.x"

If #3357986: Use the apache thats on our image instead of the php webserver. → gets merged we will need to refactor this slightly.

Follow up here.

Apache + concurrency: https://git.drupalcode.org/project/api/-/jobs/55447

Gitlab CI running times for "api" module:
* PHP built-in server (currently using this): ~30 min
* PHP bulit-in server + concurrency library: ~20 min
* Apache server (no concurrency): ~9 min
* Apache server + concurrency: ~3 min!!!

Once the apache issue ( #3357986: Use the apache thats on our image instead of the php webserver. → ) is merged, I'll adjust this one accordingly.

I already rebased the latest changes. Ready to review again.

MR: https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/32/...
Pipeline: https://git.drupalcode.org/project/gitlab_templates/-/pipelines/13563
Results:
* No concurrency: 9min 45secs - https://git.drupalcode.org/project/api/-/jobs/55852
* Concurrency: 3min 50secs - https://git.drupalcode.org/project/api/-/jobs/55840

With these changes, a new top-level variable called "_PHPUNIT_CONCURRENT" will determine whether we run the phpunit tests in parallel or not.

We are having that variable turned off by default, but we should discuss whether we should enable it by default and then, if people need all options for phpunit, then can always turn it off. The performance gain might be significant enough to decide in favor of enabling by default.

PS: there are also some minimal improvements to the artifacts we exposed that were suggested here #3372464: Improve apache usage → .

LGTM, once tests are green. I am not in favor of turning this on by default at this time. Maybe we do so after some modules start using it successfully.

Thanks for the review.

I am not in favor of turning this on by default at this time. Maybe we do so after some modules start using it successfully.

That's why I had it disabled by default and marked it as "Experimental". It's like that in the MR. When I started working on it that was exactly my thinking. Let some modules use it and if it works well, then consider turning it on by default.

I always get carried away when I see performance gains but also agree that the more conservative approach makes sense here.

Thanks for the work here! Good to put it behind a variable so folks can test without making global default.

I think this last commit (10 July 20:39 UTC) may have broken something in drupalci.variables.yml. My gitlab pipeline ran fine yesterday (10 July 15:55 UTC) but fails now with invalid yml syntax

See https://git.drupalcode.org/project/scheduler/-/pipelines/13803

I'll look into it

Thanks for fixing. Our comments overlapped.

I see that the MR pipeline run was skipped. If that had been left to run, would this have been detected?

Yup, it would have been detected, it was totally my bad.
Those text changes were a super last-minute addition and I did not check. Apologies.

No worries, you fixed it quickly. Gitlab_templates is always going to have a certain amount of "live" testing :-)

Sidenote related to #4: it seems like PHPUnit 10 is dropping support for the "--printer" option, so it makes it less of a deal I guess for the concurrent option not to have it. https://github.com/sebastianbergmann/phpunit/blob/10.0.0/ChangeLog-10.0....

Back to Fixed, as it should be.
Posting here the comment I put in slack: https://drupal.slack.com/archives/CGKLP028K/p1689066316386919
--
We added yesterday a new experimental feature to the default gitlab_templates to run phpunit tests in parallel, and hopefully reduce the total execution time of the tests.

This feature, paired with a recent change we did last week, can make the tests run 10x faster than before!!

The experimental feature is turned off by default, because we’d like people to test it out and report possible issues without disrupting others.
If you want to test it out. you just need to add _PHPUNIT_CONCURRENT: "1" under the variables section.

So your .gitlab-ci.yml file might look something like this:

include:
  - project: $_GITLAB_TEMPLATES_REPO
    ref: $_GITLAB_TEMPLATES_REF
    file:
      - '/includes/include.drupalci.main.yml'
      - '/includes/include.drupalci.variables.yml'
      - '/includes/include.drupalci.workflows.yml'

variables:
  _PHPUNIT_CONCURRENT: "1"

If you still don’t have GitlabCI in your contrib module, and you want to use it, you can follow the steps here: https://www.drupal.org/docs/develop/git/using-gitlab-to-contribute-to-dr... →

Automatically closed - issue fixed for 2 weeks with no activity.