Failing Nightwatch jobs result in a passed pipeline job

https://github.com/nightwatchjs/nightwatch/issues/2103 which points to https://github.com/mucsi96/nightwatch-api (deprecated).

Also https://github.com/mucsi96/nightwatch-api/issues/530 which suggests

we found that our issue was caused because one of the steps previous to this one that failed has an async/await statement in the callback. So, we modify it and then this step was able to fail but not break the execution and that way we were able to get the report.

None of these might be exactly our case but point to updated libraries and code refactoring.

I see that there is good progress in the related issue 📌 Update Nightwatch to 3.x Needs work so I wonder if that needs to go first as it's updating the library but also some of the tests.

Note that #3371963-23: Update Nightwatch to 3.x → points out that although the MR returns green, there are still errors thrown in the NighWatch tests, they are "just" not being reported on the top level...

It seems the main reason for these going undetected is that Nightwatch differentiates between "failures" (aka "test failed") vs "errors" (aka "something else").

The NightwatchAssertError mentioned in the IS is gone, in test runs I've done in the last few days every NightwatchAssertError I introduced led to the associated Nightwatch test failing, which in turn made the whole Nightwatch GitLab job fail.

I think it's safe to close this one.