Add Nightwatch tests

The new nightwatch test (borrowed from Core) passes in all five variants in this pipeline. At the end of the composer job we can see that 'current', 'max php' and 'next minor' use Nightwatch 3.9.0 and 'previous minor' and 'previous major' use Nightwatch 2.4.2

Next, add $BEFORE_SCRIPT_ACTIONS =~ "nightwatch-fail" to make the job fail on request.

That's odd. I only added a nightwatch: before_script to the project's .gitlab-ci.yml but we now get failures for phpcs and phpstan, when they passed 20 mins before that.

https://git.drupalcode.org/project/gitlab_templates_downstream/-/pipelin...

Seems there was something wrong with the curl execution to get the default phpcs.xml.dist
The passing job had

This project has no (.)phpcs.xml(.dist), getting default from assets/phpcs.xml.dist
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   575  100   575    0     0   4802      0 --:--:-- --:--:-- --:--:--  4831

but the failed job had

This project has no (.)phpcs.xml(.dist), getting default from assets/phpcs.xml.dist
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    12  100    12    0     0     93      0 --:--:-- --:--:-- --:--:--    93
100    12  100    12    0     0     92      0 --:--:-- --:--:-- --:--:--    93

Does this show that we only got 12 (K?) instead of the usual 575? We've had curl failures before (eg when the name is wrong) and it would be good to be able to trap that when it happens. But that's another issue. I will re-run and see what happens.

Re-run of both jobs was fine, they passed.

Here's a test from the pipeline UI, using BEFORE_SCRIPT_ACTIONS 'nightwatch-fail'
gitlab_templates_downstream-3513352/-/pipelines/450184

This is ready for review. I have also added a change to remove the SKIP_ variables as it is better to leave these to default. I know this is partly out of scope, but the SKIP_NIGHTWATCH variable was not in that list, and that's what made me think it is better to have none there.

All of the forced failed runs had

WARNING: Event retrieved from the cluster: 0/4 nodes are available: 2 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 2 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.
WARNING: Event retrieved from the cluster: 0/5 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 2 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 2 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.
WARNING: Event retrieved from the cluster: 0/6 nodes are available: 2 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 2 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/6 nodes are available: 2 No preemption victims found for incoming pod, 4 Preemption is not helpful for scheduling.

for example here. This did not seem right, but was probably unrelated, so I re-ran a couple and they ended with the normal failure, and did not have the above message. Just another random thing, I guess.

The changes look good. It makes sense to also remove the SKIP_ variables that were using the default value, so it's ok to include it in this MR.

It's the first time that I've seen that type of failure. I'm glad that re-running was enough and I'm hoping that it was a one-off, but defo worth keeping an eye on it in case it happens again.

Thanks for the review. Merged MR12 into d10-plus.

Cherry picked to d9-basic and the Nightwatch test runs ok on 'current' which is Drupal 10, but it fails in 'previous major' which is Drupal 9.5 whichs uses Nighhtwatch 1.7. The test starts OK and the user is logged in, but we then get Error: Unknown api method "textContains"
https://git.drupalcode.org/project/gitlab_templates_downstream/-/jobs/46...

There may be a way to fix this, but is it worth the trouble investigating? Do we need Nightwatch test coverage at Drupal 9? I'm inclined to just prevent this variant from running Nightwatch via a customised when: never. Is that OK?

Versions...

OLD      .assert.containsText(
NEW       .assert.textContains(

Thank you for that info. I tested d9-basic with the old function .assert.containsText and it passes on D10 and D9, however we get a deprecation warning in the D10 'current' variant, which I am not surprised about.

So instead of having the old function and deprecation, we can leave the test as-is but use nightwatch (previous major): before_script to edit the function name, and thus only change it for the specifc variant. Tested in this pipeline and both Nightwatch jobs pass with no deprecations.

There is no MR for this change, I just did it on the generic testing issue fork - here are the changes.

In addition the Nightwatch before_script there is a change to the workflow so that a push to any issue branch on 📌 Plan for initial codebase Active always triggers a pipeline, and a fix to the display of the Nightwatch version. I will align the workflow change on the other principle branches too.

This is ready for review.

Can we not check if the method exist via javascript hasOwnProperty inside the tests/src/Nightwatch/Tests/downstream-one.js file? That way we won't need the sed nor the before_script.

Untested code:

    browser
      .drupalCreateUser({
        name: 'Downy',
        password: 'Streamer',
        permissions: ['access site reports', 'administer site configuration'],
      })
      .drupalLogin({ name: 'Downy', password: 'Streamer' })
      .drupalRelativeURL('/admin/reports')
      .waitForElementVisible('body', 1000);
    if (browser.assert.hasOwnProperty('textContains')) {
        browser.assert.textContains('h1', 'Reports')
    }
    if (browser.assert.hasOwnProperty('containsText')) {
        browser.assert. containsText('h1', 'Reports')
    }
    browser.assert.noDeprecationErrors();

Thanks for that idea. I tried using browser.assert.hasOwnProperty but that gave false for each of the conditions, so no checking was done.
https://git.drupalcode.org/issue/gitlab_templates_downstream-3503337/-/p...

I then tried using typeof browser.assert.containsText === 'function' but that resulted in true for both cases.
https://git.drupalcode.org/issue/gitlab_templates_downstream-3503337/-/p...

Then after more research I found browser.expect.element().text.to.equal() and this worked in both the old and new Nightwatch variants without any conditionals.
https://git.drupalcode.org/issue/gitlab_templates_downstream-3503337/-/p...

I tidied up the code, putting it all in one statement, which I thought was going to be OK, but it wasn't. The whole thing failed, and there was also an eslint error, which must be related. 21:28 is the point in the middle of element('h1').text suggesting a line break. That was incorrect too.
https://git.drupalcode.org/issue/gitlab_templates_downstream-3503337/-/p...

So I put the browser.expect into a separate statement, which works just fine. Here are the proposed changes and this is now ready for review.

Javascript is weird sometimes... but I'm glad you found the alternative syntax. I think it's cleaner overall and that way we don't need to have the before_script to search/replace code.

So I think that the changes look good.

All done. Cherry-picked the Nightwatch change to d10-composer, just to keep the test file in line.
Also aligned the workflow changes to all other branches.

I tested via UI to make sure that the change to the javascript still works with BEFORE_SCRIPT_ACTIONS nightwatch-fail and it fails as intended, so all is good.
d9-basic https://git.drupalcode.org/project/gitlab_templates_downstream/-/pipelin...
d10-plus https://git.drupalcode.org/project/gitlab_templates_downstream/-/pipelin...

Awesome, it's great to see we have some more basic tests in the downstream projects.

In the future and when possible, when new issues are reported, we can probably build a test case "somewhere" to replicate that issue and make sure it's fixed.

Thanks for the work here.

we can probably build a test case "somewhere" to replicate that issue and make sure it's fixed.

Yes absolutely. That's what I'm going to do for 🐛 Pipeline task fails with composer.json file Active

Automatically closed - issue fixed for 2 weeks with no activity.