Update PHPUnit jobs to new driver when D11 "current" is W3C-compliant

The work previously done in MR257 removed the legacy override in the 'current' phpunit job, but adds it back only for 'previous major'. Which means that 'previous minor' 11.0 would use the new values. This is wrong, just like it was wrong for Nightwatch. The downstream failures show this - https://git.drupalcode.org/issue/gitlab_templates-3514999/-/pipelines/45...

I've pushed the changes to show the new approach, by testing the drupal core version. We have to get the actual version used, not just rely on the value of DRUPAL_CORE as this could contain composer-like constraints such as ^11.

This is working ok apart from 'next minor' 11.2, which needs some investigation.
https://git.drupalcode.org/project/gitlab_templates_downstream/-/jobs/47...

The error with 'next minor' was really simple - I forgot to add export when setting the variable, so even though it looked fine in the script, and in the after_script, it was blank in the scope of phpunit execution. The other jobs just skipped the javascript test because the webdriver was not running, but in Drupal 11.2 it caused a proper failure Driver info: driver.version: unknown which is much better. I had not noticed that the tests were skipped in the other jobs, I just saw green and assumed ok. Maybe in our downstream jobs we can add the phpunit option to fail on skipped tests _PHPUNIT_EXTRA: --fail-on-skipped

This is ready for review, however I have tested this in Scheduler and there seems to be a bit of unreliability with the new driver. The 'current' job sometimes fails, then I re-run and it passes. Here are some examples
Pass with new values scheduler-3445052/-/jobs/4746560
Failed scheduler-3445052/-/jobs/4743730
Re-run same job, get different failures scheduler-3445052/-/jobs/4746954
Re-run again and passed scheduler-3445052/-/jobs/4747136

Ready for review. It will be interesting to see if the unreliability is observed in other projects.

I wonder if this might be related (or be the source of) 🌱 [meta] Known intermittent, random, and environment-specific test failures Active .

Left feedback on the MR.

The intermittent failures could be caused by the latest Chrome release
https://github.com/teamcapybara/capybara/issues/2800

As this change now creates a new environment variable DRUPAL_CORE_VERSION which is generic, is created in the composer job and can be used anywhere, do you think that this work should really be done in a separate issue, with a separate entry in the changelog.md? It would be done first, then return here when merged, just for the phpunit driver changes. I'm OK to leave as-is and do all in the same MR because it is not a very complicated change, (and it also reduces work for both of us) but just thought I would check with you.

I'm happy either way. On one hand it makes sense two issues as they are easy to isolate and it'll be clear what the changes are, but on the other we only created that new variable because we needed it for this issue, so it's a kind of nice side effect of it.

I'd say let's leave it all in one issue, as it doesnt' really need extra documentation.

I suggested a name change as it was too close to DRUPAL_CORE.

Good, let's keep it all here. I have renamed the variable to INSTALLED_DRUPAL_VERSION as suggested in the MR.

I note that there are some temporary changes to undo in the downstream jobs definition.

Would it be a good idea to also create this new variable in the D7 pipeline? It won't be quite the same becauase Drupal is not installed until the phpunit jobs. There is no specific need for it yet in D7, this issue has no other changes for D7, so maybe not worth the effort.

Regarding Drupal 7 I have answered my own question, and we do not need to do it. There are only two variants, both using the same D7 core version. Technically it would be possible to add other variants to test earlier D7 releases but that is pointless. So there is no need for the new variable.

This is ready for review. Could you trigger the other downstream pipelines? I've made reminders in the MR to remove the extra test lines.

Yeah, agree that we don't need anything for D7.

I made a question in the MR about a small refactoring, but I'm not sure if it's like that intentionally. NW based on that question and possible refactoring.

I have made the change to create the new variable MINK_DRIVER_ARGS_WEBDRIVER_DEFAULT and I think this does simplify the readability and understandability of the change.

Tested in the downstream pipelines and all OK. Testing with more complex tests (eg Scheduler) is still producing variable and unpredictable results. I ran a job and it failed, then I re-ran it in the same pipeline with no other changes and the failures were different. These tests have passed with the new webdriver before, and I expect they will again. Could you trigger the other downstream pipelines please? I will also runa test with a customized override to make sure that is still catered for.

I don't know if this helps, but attached is a comparison of the logs of the different failure on re-running. This is one particular test, and the first run shows "element not found" but the second says "stale element reference".

I have made a futher change, that is both more efficient and also gets us better information. I noticed that .show-context was getting the real \Drupal::VERSION via vendor/autoload.php and this snippet was being called from several jobs. So instead, I have replaced the composer show with that php -r autoload.php and store the variable that way. Then when *show-context is called we simply write the value out, instead or running the php command. The value of DRUPAL_CORE is only displayed when it differs from the actual installed version. We can see the benefit of this in this next minor job where we do now get 11.2 showing.

Those are nice additions too, thanks! Also, the refactoring of the variable makes the comment easy to follow, so I'm happy with that.

Rest of downstream pipelines: https://git.drupalcode.org/issue/gitlab_templates-3514999/-/pipelines/46...

Re #14, not sure what can be the reason for the mixed results. I think core suffers from similar things. I don't think that it's anything new that we might be introducing here, but something related to the driver service.

Once you've run the last round of tests that you mention, I think we can remove the debug code and this should be good to go. Great job!

Good that you like the last changes, I was very happy when I discovered how to make it both better, clearer, and more efficient all at the same time :-)

Here's a Scheduler test overriding the 'current' job with the legacy values
'show environment variables' has all 3 variables as expected, and the old driver values are used.

I'll remove all the temporary test code now.

All temporary code removed. This is ready now.

Great, this is ready. RTBC. Thanks!

This is now merged.

Thanks. It's been a long time coming, both this issue and it's sibling 📌 Update to new driver when D11 "current" uses Nightwatch 3 Postponed
I'm pleased we found a good solution for the new INSTALLED_DRUPAL_VERSION variable.