Update Nightwatch to 3.x

We are having trouble upgrading Nightwatch past 2.4.2 already, see 📌 Update Nightwatch from 2.4.2 to 2.6.19 Closed: duplicate - whatever is broken there needs fixing first.

I was able to get nightwatch 3.0.1 running and have submitted a merge PR. But several tests were failing. I am trying to setup a local environment to try and debug the tests, till now I have not been able to setup a test environment.

longwave → closed merge request !4370

The domain setting in cookies does not work the same way when w3c mode is enabled, removing it from the SIMPLETEST_USER_AGENT cookie works locally for me.

longwave → credited effulgentsia → .

longwave → credited Spokje → .

Crediting contributors to 📌 Update Nightwatch from 2.4.2 to 2.6.19 Closed: duplicate

DrupalCI doesn't seem to pick up on the five NightwatchAssertErrors I see in the full test log.

Adding Needs followup for that.
Even whilst the full focus is, rightfully so, on the GitLab move, having a green TestBot whilst there are errors is bad.

00:40:15.270   ✖ NightwatchAssertError
00:40:15.270    Timed out while waiting for element <[href*=edit-editor-settings-plugins-ckeditor5-sourceediting]> to be present for 5000 milliseconds. - expected "visible" but got: "not found" (5082ms)
00:40:15.270 
00:40:15.270     Error location:
00:40:15.270     /var/www/html/core/modules/ckeditor5/tests/src/Nightwatch/Tests/ckEditor5CodeSyntaxTest.js:
00:40:15.270     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:40:15.270      36 |         .keys(browser.Keys.DOWN) // Hit the down arrow key to move it to the toolbar.
00:40:15.270      37 |         // Wait for new source editing vertical tab to be present before continuing.
00:40:15.270      38 |         .waitForElementVisible( 
00:40:15.270      39 |           '[href*=edit-editor-settings-plugins-ckeditor5-sourceediting]',
00:40:15.270      40 |         )
00:40:15.270     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:40:15.391   FAILED: 1 assertions failed and  5 passed (8.962s)


00:40:26.978    Timed out while waiting for element <[href*=edit-editor-settings-plugins-ckeditor5-sourceediting]> to be present for 5000 milliseconds. - expected "visible" but got: "not found" (5088ms)
00:40:26.978 
00:40:26.978     Error location:
00:40:26.978     /var/www/html/core/modules/ckeditor5/tests/src/Nightwatch/Tests/ckEditor5EditorHeightTest.js:
00:40:26.978     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:40:26.978      38 |         .keys(browser.Keys.DOWN) // Hit the down arrow key to move it to the toolbar.
00:40:26.978      39 |         // Wait for new source editing vertical tab to be present before continuing.
00:40:26.978      40 |         .waitForElementVisible( 
00:40:26.978      41 |           '[href*=edit-editor-settings-plugins-ckeditor5-sourceediting]',
00:40:26.978      42 |         )
00:40:26.978     ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:40:26.978 
00:40:27.196 
00:40:27.196   FAILED: 1 assertions failed and  5 passed (9.067s)

00:41:54.180   ✖ NightwatchAssertError
00:41:54.180    Failed [ok]: (The expression evaluated to a falsy value:
00:41:54.180 
00:41:54.180   assertModule[propName].apply(null, this.args)
00:41:54.180 ) - expected "true" but got: "false" (0ms)
00:41:54.180 
00:41:54.180     Error location:
00:41:54.180     /var/www/html/core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroMobileMenuTest.js:
00:41:54.180     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:41:54.180      117 |         [linkSubMenuId],
00:41:54.180      118 |         (result) => {
00:41:54.180      119 |           browser.assert.ok(result.value); 
00:41:54.180      120 |         },
00:41:54.180      121 |       )
00:41:54.180     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:41:54.180 
00:41:54.334 
00:41:54.334   FAILED: 1 assertions failed and  4 passed (1.001s)


00:42:50.917   ✖ NightwatchAssertError
00:42:50.917    Timed out while waiting for element <.block-search-wide__wrapper> to not be visible for 5000 milliseconds. - expected "not visible" but got: "visible" (5114ms)
00:42:50.917 
00:42:50.917     Error location:
00:42:50.917     /var/www/html/core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js:
00:42:50.917     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:42:50.917      81 |       .keys(browser.Keys.TAB)
00:42:50.917      82 |       .pause(50)
00:42:50.917      83 |       .waitForElementNotVisible(searchWideSelector) 
00:42:50.917      84 |       // Release all keys.
00:42:50.917      85 |       .keys(browser.Keys.NULL);
00:42:50.917     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:42:50.917 
00:42:51.081 
00:42:51.081   FAILED: 1 assertions failed and  3 passed (5.959s)

00:43:03.680   ✖ NightwatchAssertError
00:43:03.680    Testing if element <button.sticky-header-toggle> is visible in 5000ms - expected "is visible" but got: "not visible" (5191ms)
00:43:03.680 
00:43:03.680     Error location:
00:43:03.680     /var/www/html/core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroStickyHeaderToggleTest.js:
00:43:03.680     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:43:03.680      21 |       .assert.attributeEquals(buttonSelector, 'aria-checked', 'false')
00:43:03.680      22 |       .getLocationInView('footer.site-footer', () => {
00:43:03.680      23 |         browser.assert.visible(buttonSelector); 
00:43:03.680      24 |         browser.assert.not.visible('#site-header__inner');
00:43:03.680      25 |       })
00:43:03.680     –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
00:43:03.680 
00:43:03.833 
00:43:03.833   FAILED: 1 assertions failed and  2 passed (5.753s)

Also, we can now drop our workaround with the "resolutions"-section in core/package.json, since nightwatch 3.1.0 has a "safe" version of the semver-package.

I don't know if there are any differences between Chrome 106 and Chrome 115 that are relevant to this issue, but FYI that I just now opened #3377509: Update Chrome container to use newer version (115 or higher) → .

oliveroStickyHeaderToggleTest fails due to getLocationInView() not working now we are in w3c mode: https://github.com/nightwatchjs/nightwatch/issues/3091

We will need to find a workaround for that.

The other issues are due to sending special keys such as Tab or arrow-down - this also no longer seems to work, the driver responds with

{
     value: {
       error: 'unknown command',
       message: 'unknown command: Cannot call non W3C standard command while in W3C mode',
       stacktrace: ''
     }
  }

Whoops, got to the same solution as @longwave, but was trying to post it in my [ignore]-patch issue.

I think we have another candidate follow-up issue:

core/tests/Drupal/Nightwatch/Tests/claroAutocompleteTest.js passes, but the autocomplete JavaScript used in the test seems to be 404?
https://dispatcher.drupalci.org/job/drupal_patches/195337/artifact/jenki...(Coordinated-Universal-Time)_console.json

So, per usual, @longwave found the correct way.

Also, per usual, I stumble over the weirdest stuff:

Commit 8ebeb952 has only one error showing in the full CI log, yet looking at the result artifacts made by nightwatch, we have 5 failures.

I strongly suspect any Nightwatch-test ending with a

(result) => {
            browser.assert

will not show up in the log when it fails.

It's seems we're _really_ want to make it hard to see a nightwatch failure :/
Yet Another Follow-Up, me thinks.

Not sure what's happening with the Nightwatch output, can't reproduce this locally - yarn exits with a non-zero return code which should indicate failure, but on DrupalCI it just seems to abort and truncate the log.

So if we dig really deep (https://dispatcher.drupalci.org/job/drupal_patches/195437/artifact/jenki...), we see that there are 2 remaining failing tests.

The fact that we have really deep, since:
1) TestBot is green anyway
2) You have to look at the full log to see any reported fail in the log
3) Which then turns out to not even show all the fails
leaves me wondering if there's any use of a test subsystem that is giving a false sense of security and most probably have been doing so since it introduction.

Add to that we didn't seem to mind too much when it couldn't be updated when 9.5/10.0 came out, nor when 10.1 was released, is making me feel Nightwatch-testing isn't getting same amount of attention/love the rest of core is getting.

For me, it's a reason to spend my time elsewhere in the queue, I think it's not worth putting time in upgrading a test system that's "green" anyway, unless you really, really spend time to find a failure that is big enough to show a glimpse of the fail showing up in the top reporting layer.

it needs CR to point changes in tests

Opened 🐛 Failing Nightwatch jobs result in a passed pipeline job Active but agreed with #23 it's not clear how useful the nightwatch suite is if it's been giving us false negatives for years.

This is also blocking further progress on 📌 [PP-2] Speed up gitlab ci runs Postponed now since after the changes on that issue, nightwatch becomes the joint-slowest test group.

Can we skip the failing tests and open a follow up issue for each (or all) of them?

https://nightwatchjs.org/guide/running-tests/skipping-disabling-tests.html

@mstrelan yeah I think that's reasonable, otherwise any new nightwatch tests we add could also end up blocking this issue too.

It took me a very, very long time to get Nightwatch tests to pass on GitLab CI for a contrib module I maintain.

Turns out the root cause is that the d.o GitLab template uses the Nightwatch test runner in a very good way but sadly broken way: 🐛 Impossible to run only Nightwatch tests in a given directory (f.e. for contrib modules) Needs review .

leaves me wondering if there's any use of a test subsystem that is giving a false sense of security and most probably have been doing so since it introduction without being noticed by anybody since it introduction in 2018.

I've noticed this too in the past few years, occasionally, specifically while working on Nightwatch tests. I think the very low number of Nightwatch tests in Drupal core is an important factor in how this has slipped through the cracks. Another important factor seems to be the less-than-awesome mapping of DrupalCI "number of tests" (and failures) to actual reports. (Meaning: it's AFAICT not actually limited to Nightwatch tests.)

Hopefully the transition to GitLab CI will make things more precise, standardized and reliable!

I had a go at rebasing this. There were conflicts in the following files and I may have not done it correctly. Nightwatch tests do not complete. They hang at installProfileTest.js. See https://git.drupalcode.org/project/drupal/-/jobs/770136

• core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroSearchFormTest.js
• core/tests/Drupal/Nightwatch/Tests/Olivero/oliveroMobileMenuTest.js
• core/modules/ckeditor5/tests/src/Nightwatch/Tests/ckEditor5CodeSyntaxTest.js
• core/modules/ckeditor5/tests/src/Nightwatch/Tests/ckEditor5EditorHeightTest.js