Nightwatch tests do not work with the default template: port the approach from core

The set up is a bit more complex in this other project https://git.drupalcode.org/project/gitlab_ci_testbed_for_drupal_core/-/b... but that one is working for core.

We should be able to borrow some of the configuration and/or needed variables from there.

I can confirm that the nightwatch job in the d.o GitLab template does not work at all.

I spent 1–2 days on getting it to work in https://git.drupalcode.org/project/acquia_migrate/-/merge_requests/2. And now over at 📌 Adopt GitLab CI Needs review , I'm finding myself having to fight it again.

The nightwatch job was AFAICT only ever tested with https://git.drupalcode.org/project/keycdn/-/blob/8.x-1.x/tests/src/Night... AFAICT, and that tests … very little 😅

I had to port the nightwatch stuff from Drupal core's GitLab template to get it to work (from 📌 GitLab CI integration for core Needs work ). Which then in turn forced me to modify the artifacts key's values.

That also led to me discovering 🐛 Impossible to run only Nightwatch tests in a given directory (f.e. for contrib modules) Needs review , which we fortunately can work around.

I'm currently working on adopting GitLab CI for the Decoupled Pages module, which uses only Nightwatch tests. That's why I'm running into this once again at 📌 Adopt GitLab CI Needs review . Stay tuned!

This 👆 has proven to work on: 9.4.x, 9.5.x, 10.0.x, 10.1.x, 10.2.x and 11.x, all simultaneously: https://git.drupalcode.org/project/decoupled_pages/-/pipelines/40238

See the changes I made to the nightwatch job's script there, and note that it's identical to what I'm proposing here.

Please test! 🙏

Granted @fjgarlin & @moshe weitzman VCS access and project editing access to https://www.drupal.org/project/decoupled_pages → , so they can use it as a test bed in the future for all Nightwatch-related testing changes. That way, the keycdn example (https://git.drupalcode.org/project/keycdn/-/blob/8.x-1.x/tests/src/Night...) could theoretically be retired.

This blocks #3397270: [PP-2] Nightwatch testing against all opted in versions → .

One thing that decoupled_pages doesn't do, which was the original title for this issue, is do drupalLogin/drupalLogout. We shouldn't lose sight of that in the issue

I think to solve that we need to be able to grab the php logs so we can get that full stack trace. I'm not entirely how to do that myself.

MR that adds Decouples Pages to this project's testing is at https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/71. No corresponding d.o. issue.

Making sure that #12 is addressed. Once the main issue is in place, we can also merge #13 (thanks!).

I'm not sure exactly what changes the decoupled_pages template is doing based on core, but it does solve the drupalLogin problem as well.

I tried a modified version here: https://git.drupalcode.org/project/cck_select_other/-/pipelines/41694

Simplified version from decoupled_pages:

nightwatch:
  # Run tests with `--tag=$CI_PROJECT_NAME`, otherwise we get a `The command .clickOperation() is already defined.` error.
  script:
    - !reference [ .setup_webserver ]
    - !reference [ .simpletest_db ]
    # Provide some context on the test run.
    - vendor/bin/drush status
    - export DRUPAL_TEST_DB_URL=$SIMPLETEST_DB
    - cd $_WEB_ROOT
    # @see https://git.drupalcode.org/project/drupal/-/commit/dfa16b8aca8e92be1f00a021c520a645f11f0227 — but tweaked to generate a correct JUnit report
    - cp ./core/.env.example ./core/.env
    # dotenv-safe/config does not support environment variables
    # @see https://github.com/rolodato/dotenv-safe/issues/126
    # @todo move this to `variables` when the above is resolved
    - echo "DRUPAL_TEST_BASE_URL='http://localhost/$_WEB_ROOT'" >> ./core/.env
    - echo "DRUPAL_TEST_CHROMEDRIVER_AUTOSTART=false" >> ./core/.env
    - echo "DRUPAL_TEST_DB_URL='${DRUPAL_TEST_DB_URL}'" >> ./core/.env
    - echo "DRUPAL_TEST_WEBDRIVER_HOSTNAME='localhost'" >> ./core/.env
    - echo "DRUPAL_TEST_WEBDRIVER_CHROME_ARGS='--disable-dev-shm-usage --disable-gpu --headless'" >> ./core/.env
    - echo "DRUPAL_TEST_WEBDRIVER_PORT='9515'" >> ./core/.env
    - cat ./core/.env
    - mkdir -p ./sites/simpletest ./sites/default/files /var/www/.cache/yarn /var/www/.yarn ./core/reports
    - chown -R www-data:www-data ./sites/simpletest ./sites/default/files /var/www/.cache/yarn /var/www/.yarn ./core/reports ./core/.env
    # @todo Uncomment this line and remove the line after it once https://www.drupal.org/project/drupal/issues/3389763 lands.
    #- sudo BABEL_DISABLE_CACHE=1 DRUPAL_NIGHTWATCH_SEARCH_DIRECTORY=modules/custom/ -u www-data yarn --cwd ./core test:nightwatch
    - sudo BABEL_DISABLE_CACHE=1 -u www-data yarn --cwd ./core test:nightwatch --tag=$CI_PROJECT_NAME
  after_script:
    - cp /var/log/apache2/test.apache.access.log $CI_PROJECT_DIR/apache.access.log.txt
    - cp /var/log/apache2/test.apache.error.log $CI_PROJECT_DIR/apache.error.log.txt
  artifacts:
    expire_in: 6 mos
    expose_as: 'reports'
    when: always
    reports:
      junit: $_WEB_ROOT/core/reports/nightwatch/Tests/*.xml
    paths:
      - $_WEB_ROOT/core/reports/nightwatch
      - $CI_PROJECT_DIR/apache.access.log.txt
      - $CI_PROJECT_DIR/apache.error.log.txt

Thanks for the example. I guess also keeping both keycdn and the new decoupled_pages will ensure more coverage.
The pipeline fails though with keycdn in the nightwatch job: https://git.drupalcode.org/project/keycdn/-/jobs/253538

@mradcliffe in #15:

I'm not sure exactly what changes the decoupled_pages template is doing based on core, but it does solve the drupalLogin problem as well.

Yay! 😊 I was 99% confident this would be the case. Great to see it confirmed by others.

@fjgarlin The reason keycdn is failing is because it has the wrong directory structure:

https://git.drupalcode.org/project/keycdn/-/blob/8.x-1.x/tests/src/Nightwatch/exampleTest.js

👆 is tests/src/Nightwatch/*.js

Quoting core/tests/README.md:

* Nightwatch will run tests for core, as well as contrib and custom modules and
  themes. It will search for tests located under folders with the pattern
  `**/tests/**/Nightwatch/(Tests|Commands|Assertions)`

and

Nightwatch tests, as well as custom commands, assertions and pages, can be
placed in any folder with the pattern
`**/tests/**/Nightwatch/(Tests|Commands|Assertions|Pages)`. For example:
```
tests/Nightwatch/Tests
src/tests/Nightwatch/Tests
tests/src/Nightwatch/Tests
tests/Nightwatch/Commands
tests/src/Nightwatch/Assertions
tests/src/Nightwatch/Pages
```

Conclusion

keycdn should do:

mkdir tests/src/Nightwatch/Tests
git mv tests/src/Nightwatch/exampleTest.js tests/src/Nightwatch/Tests/
git commit -m 'Use the directory structure for Nightwatch tests that Drupal core requires.'

Thank you @Wim Leers and everyone. Catching up here and I can confirm that these changes fix the issue I reported here originally: https://git.drupalcode.org/project/ckeditor5_paste_filter/-/jobs/265709

_GITLAB_TEMPLATES_REF = 3375359-nightwatch
_GITLAB_TEMPLATES_REPO = issue/gitlab_templates-3375359

I tested by updating the CI variables on GitLab, which required a bit of digging and I think the README for this project needs a minor touch-up. I will be creating a small issue and MR for that.

Nice, I' had also been wondering about that — that'll be a great addition to the docs, @star-szr 🤩

Here is that issue: #3398680: Incorrect instructions for testing issue forks in README →

I can't speak as much to the code review aspect (though it does look good to me), but this works great, so marking as RTBC!

Merged https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/70

Going to add now too https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/71

Big-time improvement!! Thanks so much again.

MR71 (adding decoupled_pages) didn’t go well: https://git.drupalcode.org/project/gitlab_templates/-/pipelines/43390

I reverted the change for now https://git.drupalcode.org/project/gitlab_templates/-/commit/10ba834feba... to make sure we’re not breaking things for other people.

The root cause for that is #3398716-8: Optimize `composer-lint`, `phpcs`, `stylelint` and `eslint` jobs → . Now fixed. 👍

I realized too late that I failed to include the fix for the reports 🙈 This is why e.g. https://git.drupalcode.org/project/ckeditor5_paste_filter/-/pipelines/43... does not list the Nightwatch test results!

New one-line MR for that: https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/79/...

I did not know to check for that! I will run another test using https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/79

Ahh, much better: https://git.drupalcode.org/project/ckeditor5_paste_filter/-/pipelines/43...

Perfect! Thanks for the quick fix and check too.

Automatically closed - issue fixed for 2 weeks with no activity.