Test Webform against upcoming 10.2.x on GitLab CI — and use concurrency to make tests 6x faster (on GitLab CI)

Done. Note that we could go much further — see https://git.drupalcode.org/project/big_pipe_sessionless/-/blob/2.x/.gitl... for example:

  # Opt in to testing current minor against max supported PHP version.
  OPT_IN_TEST_MAX_PHP: '1'
  # Opt in to testing previous & next minor (Drupal 10.0.x and 10.2.x).
  OPT_IN_TEST_PREVIOUS_MINOR: '1'
  OPT_IN_TEST_NEXT_MINOR: '1'
  # Opt in to testing $CORE_PREVIOUS_MAJOR (currently Drupal 9.5).
  OPT_IN_TEST_PREVIOUS_MAJOR: '1'
  # Opt in to testing $CORE_MAJOR_DEVELOPMENT (currently Drupal 11).
  OPT_IN_TEST_NEXT_MAJOR: '1'

This is awesome!!!

Do you want me to add #3 too? :)

I didn't know about that flag, that is neat, BUT: The regular phpunit job on GitlabCI currently takes *30 minutes* for webform, because the contrib template currently uses zero concurrency/optimization unlike core and this module has a *lot* of tests.

That makes running *every* pipeline against one or even multiple extra core versions very expensive.

It is possible to manually run a pipeline against 10.2.x, that's what I did for 🐛 [PP-2] to_email and similar config keys have schema type email but allow other things Needs work where I linked those results as well.

Not a maintainer, but IMHO it would be better to do it like core does, where alternative phpunit jobs are available on every MR to be started manually, because doing it through the UI as with custom variables is tedious.

The regular phpunit job on GitlabCI currently takes *30 minutes* for webform

🤯

Yeah we should improve that so that it is improved for all contrib modules. But isn't it as simple as setting _PHPUNIT_CONCURRENT: "1" per #3370952: Run phpunit tests from a single job in parallel → ? Let's find out.

IIRC I tried that and at least back when I tried that, that mode didn't suppress deprecations.

@Berdir fixed that using https://git.drupalcode.org/project/webform/-/merge_requests/380/diffs?co... 😊 (I spent far too long on that exact problem over at https://git.drupalcode.org/project/acquia_migrate/-/merge_requests/2, but I did find a solution at least!)

Thanks to @Berdir for nudging me in this direction 😇

Off topic thoughts:

Ah, the extra trick is nice. and the performance improvement is impressive. Good to know. FWIW, this should be the default configuration for contrib IMHO, because I don't really want to adapt my 100 projects to have this overridden and risk breaking something in the future (sure, most contrib projects do not have remotely as many tests as webform). There's the meta issue about backporting core improvements to contrib, should be added there I think?. I assume it's OK to update the template and change how contrib tests work by default.

(Difference in output is due to switching from sequential to parallel test running per @Berdir in #6.

Conclusion

@alexpott was right in #3402168-12: Follow-up for #3361534: Config validation errors can still occur for contrib modules, disrupting contrib → : the core MR does not yet fix it for functional tests.

Since #3402168-17: Follow-up for #3361534: Config validation errors can still occur for contrib modules, disrupting contrib → , this is passing all of the 10.2.x tests 👍

With the exception of unrelated failures (for example, WebformStatesHiddenTest and WebformSettingsPreviewTest fail because it's asserting markup literally instead of using CSS selectors, XPath or DOMDocument — probably these output changes are due to 🐛 Upgrade filter system to HTML5 Fixed ). See the results.

Note: The alternative version this extends from was AFAIK merged and the patch committed, so this will need to be update to remove those things.

@Wim Leers & @Berdir Thank you for helping set this up.

The most immediate thing I did to help move this forward was to disable the old DrupaCI tests.

The fact that the 10.1.x tests are passing, and 6x faster is excellent. I am a little thrown off by the 10.2.x test failures but I am comfortable dealing with that in another ticket.

> The most immediate thing I did to help move this forward was to disable the old DrupaCI tests.

Note that you then don't get testing of patches anymore.

DrupalCI isn't slow. GitlabCI without the changes here is slow. Now with those changes, GitlabCI is likely a bit faster than DrupalCI, not sure how much, but should not make a huge difference.

@Berdir Good catch. I restored the issue and commit test.

To @Berdir's point: tweaking the title! :)

FYI: the DA will turn off DrupalCI at some point. But I agree that for a project like this, it may be better to just wait until GitLab CI contrib has been optimized further.

Also unassigning.