Nightwatch "next major" failing

Turns out this runs on 11.0.4 just fine, but fails on 11.0.x - these are the only 6 commits since:

• #3474108 → by smustgrave, tomomi: Replace eslint-plugin-jquery with eslint-plugin-no-jquery
• #3465154 → by yujiman85, n-m-daz, nod_: Update run-tests.sh help output to match current test organization
• #3470114 → by quietone, smustgrave: Change string 'Modules to enable' that are not in comments
• #3471949 → by mstrelan: Allow passing MarkupInterface to AssertContentTrait::setRawContent
• #3473214 → by mstrelan: Stop passing ints to DateTimePlus::createFromFormat and DrupalDateTime::createFromFormat
• #3302833 → by akhil babu, b_sharpe, alexpott, oily, smustgrave: Improve PluginNotFound exception to include possible shorthand action IDs

That might be wrong (above). I was using drupal-mrn.dev but more has been committed to 11.x-dev. I think we'll need to try and git bisect and run the tests and try to figure out which commit caused the issue.

Turns out this runs on 11.0.4 just fine, but fails on 11.0.x

If you wanted to prove this in the pipeline, you can add:

composer (next major):
  variables:
    DRUPAL_CORE: '11.0.4'

to temporarily fix the core version. It would be good to see the nightwatch test pass in a pipeline here. Then we know exactly what we are dealing with.

That was interesting. I ran it with core 11.0.4 at the Nightwatch version was 2.4.2 not 3.7.0 which we get with 11.x-dev
https://git.drupalcode.org/issue/project_browser-3477343/-/jobs/2883174

I've done another test, using 11.0.x-dev for the Drupal core version in 'Next Major'. This shows the actual versions is 11.0.5-dev in the Nightwatch job, and as that is running Nightwatch 2.4.2 it also needs the legacy driver. The tests end green
https://git.drupalcode.org/issue/project_browser-3477343/-/pipelines/296863

To me, this shows that the problem is caused by the differences between Core 11.0.x-dev / Nightwatch 2.4.2 and Core 11.x-dev / Nightwatch 3.7.0. Apologies is that was obvious, but I wanted to check the comment in #4 "but fails on 11.0.x" as that is not strictly true. Hope this helps. You can revert my last commit when you next want to test at 11.x-dev

Oh wonderful, I think this is actually what we want - the 11.0.x-dev to be our test base. Really appreciate you jumping in, @jonathan1055!

Pleased to help.
What about your PHPUnit jobs? They currently pass at 11.x-dev with the new selenium driver
https://git.drupalcode.org/project/project_browser/-/jobs/2882356

So the question is, do you want those to stay at 11.x-dev or bring them back also to align with Nightwatch at 11.0.x-dev? Let me know you preference and I can adjust your template, becauase temporarily I skipped phpunit (knowing it would fail the way I set up the demonstration of Nightwatch)

There have been a few changes in Gitlab Templates since 1st Ocober. In particular 📌 Update templates so 11.0 is the default/current branch RTBC which was on 8th October. I will rebase the MR and then you can decide how to proceed, as there is no 'next major' job at the moment.

Now that Drupal 11 is current, you probably want to also test with Drupal 10, so I will add the opt_in for that. There is no 'Max PHP' variant at the moment, either, because D11 only supports PHP8.3, it is not ready for PHP8.4 yet. That's why this pipeline has fewer jobs that before. You can leave OPT_IN_TEST_MAX_PHP:1 and as soon as core supports PHP8.4 the variant will be added to the pipeline.

Am I right in thinking that your original two commits on this MR were invesitgation into why the Nightwatch tests failed, and that these changes can be removed now? There were conflict, and also now eslint failures (double to single quotes), but I think all the changes to tests/src/Nightwatch/Tests/keyboardTest.js can be reverted? I won't do that until you confirm.

I've rebased this issue against the latest 2.x changes.

There were a number of changes to the UI that affected the Nightwatch tests, including changes to the labels and values in the category drop-down, as well as changing the project browser route to include the source plugin.

As the new route can take the browser directly to the project-browser-test-mock source, I've taken out the config step that was disabling other sources.

I'm still running the older Nightwatch version on my local, so I'll do some more on this as soon as I figure out how to upgrade it.

I've had some success at last.

I can run Nightwatch locally, but it doesn't seem to be running with the same settings as in GitLabCI, as the tests are failing on things that pass on GitLabCI.

However, I have been able to get enough information from the local runs to fix the tests so they run in GitLabCI.

The main issue seems to be the browser.keys() call, which is deprecated, and I was using to send keypresses. I have replaced with browser.perform(), and send the appropriate key in the callback function as a user action.

The next step is to rebase for the recent changes, and add back the tests that were disabled for this issue.

Nightwatch tests now working for all current versions.

Jonathan had enabled "previous major", so I tried leaving that in, but one of the FunctionalJavascript tests was failing, and fixing that would be outside the scope of this change, so I turned it off.

Aaaand we're back!

Automatically closed - issue fixed for 2 weeks with no activity.