Test-only changes job tries to run when Nightwatch test files are changed

The test-only job rule is:

    - changes:
        - tests/**/*
      when: manual

This needs to include tests/src/Functional, tests/src/FunctionalJavascript, tests/src/Kernel and tests/src/Unit but not tests/src/Nighwatch.

Is it too restrictive to include those four directories explicitly? Maybe it's OK, because 'test-only changes' can be documented as only working with the standard directory-naming structure, and if a project has tests elsewhere?

The PHPUnit jobs use the following .phpunit-tests-exist-rule

  - exists:
      - '**/tests/**/*Test.php'
    when: on_success

So maybe the test-only rule could be changed to be

    - changes:
        - tests/**/*Test.php
      when: manual

I notice also that the test-only rule does not have the leading '**' and maybe that should also be added, to find tests in sub-modules too.

Yeah I think we can follow the same expression as the phpunit-tests-exist-rule rule.

Tested in this pipeline with default-ref (not this MR) to check that the test-only jobs runs when it is should not. The problem is that it did not run. The repo does not have any nightwatch tests but the MR was adding one. The composer log shows:

$ git show -2 --stat --oneline
cc9a4dc Merge branch 'mr292-3488104-test-only-ignore-nightwatch' into '2.x'
 .gitlab-ci.yml               | 60 +++++++++++++++++++++++++++++++++++++-------
 tests/src/Nightwatch/temp.js |  1 +
 2 files changed, 52 insertions(+), 9 deletions(-)
03a8821 Baseline without using MR292
 .gitlab-ci.yml               | 60 +++++++++++++++++++++++++++++++++++++-------
 tests/src/Nightwatch/temp.js |  1 +
 2 files changed, 52 insertions(+), 9 deletions(-)

It seems that the existing rule changes: - tests/**/* does not trigger when the change is adding a new file. It would be good to make that work properly, because the concept of 'test-only changes' should cater for new tests, not necessarily just changes to existing tests.

Yeah, it makes sense that changes: doesn't include newly added files as those would actually be tested in the correct jobs (either PHPUnit or nightwatch), so maybe we don't need to worry about it.

When adding a new test as proof that a bug fix has coverage that would not be shown in the normal phpunit tests, becuase the modified/fixed code would also be included. So it would have been helpful if new fles did trigger the job. But as they don't, there not a lot we can do. I will test on decoupled_pages instead.

It was my error - newly added test files do trigger the 'test-only changes' job, which is good. I had OPT_IN_TEST_CURRENT: 0 which meant that the job was not added, regardless of the changes:

Existing behavior without MR292 on Scheduler, when a nightwatch .js test is added. The 'test-only changes' job is run when it should not.
https://git.drupalcode.org/project/scheduler/-/pipelines/343379

Using MR292 - the 'test-only chnages' job is correctly not added
https://git.drupalcode.org/project/scheduler/-/pipelines/343391

With MR292 and also making a change to a PHPUnit test - the job is added as required
https://git.drupalcode.org/project/scheduler/-/pipelines/343395

I also tested with Decoupled Pages, which has Nightwatch tests in the repo. The MR makes a change to the .js test file
"before" test shows the 'test-only changes' job is incorrectly added to the pipeline
https://git.drupalcode.org/issue/decoupled_pages-3422594/-/pipelines/343168

"after" using MR292 shows that the 'test-only changes' job is not added
https://git.drupalcode.org/issue/decoupled_pages-3422594/-/pipelines/343244

This is ready for review.

Oh cool, even better that it detects new files!

Thanks for the testing. Given that the code changes are really minor and the tests included in #8, this is RTBC.

Merged, thanks!

Automatically closed - issue fixed for 2 weeks with no activity.