Tests failing

Bingo ! :-)

Okey let me wrap it up and share as much details I can, as it was quite intensive research:

1. This is how drupalci runs javascript tests:

23:47:56 cd /var/www/html && sudo MINK_DRIVER_ARGS_WEBDRIVER='["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-gpu","--headless"]}}, "http://chromedriver-jenkins-drupal-contrib-654786:9515"]' -u www-data php /var/www/html/core/scripts/run-tests.sh --color --keep-results --suppress-deprecations --types "PHPUnit-FunctionalJavascript" --concurrency "1" --repeat "1" --sqlite "/var/lib/drupalci/workdir/run_tests.js/simpletest.sqlite" --dburl "mysql://drupaltestbot:drupaltestbotpw@172.18.0.4/jenkins_drupal_contrib_654786" --url "http://php-apache-jenkins-drupal-contrib-654786/subdirectory" --directory modules/contrib/simple_recaptcha

2. In order to reproduce the errors locally, 2 pieces were needed:

First of all run chromedriver with bare minimum settings and args, here's my docker-compose template:

version: '3.5'
services:
  chrome:
    image: drupalci/webdriver-chromedriver:production
    ulimits:
      core:
        soft: -1
        hard: -1
    cap_add:
      - SYS_ADMIN
    volumes:
      - /dev/shm:/dev/shm
    ports:
      - "4444:4444"
    entrypoint:
      - chromedriver
      - "--port=4444"
      - "--allowed-ips="
      - "--allowed-origins=*"
    networks:
      - default

Second thing, the MINK_DRIVER_ARGS_WEBDRIVER in command above is being passed along to the getMinkDriverArgs():
https://git.drupalcode.org/project/drupal/-/blob/10.1.x/core/tests/Drupa...
So we'd like to keep this one same locally, here's line for phpunit.xml file:

<env name="MINK_DRIVER_ARGS_WEBDRIVER" value='["chrome", {"browserName":"chrome","chromeOptions":{"args":["--disable-gpu","--headless"]}}, "http://chrome:4444"]'/>

3. Now we are able to reproduce the tests locally with quite close set up

Tests are failing only for JS tests, and it seems to be a problem with recaptcha iframe,
the key here is that we connect to chrome via insecure connection:
http://chrome:4444

drupalci as well:

http://chromedriver-jenkins-drupal-contrib-654786:9515

This is causing issues with recaptcha iframe, getting into cross origin issues and blocking some js from recaptcha domains ( those are loaded via https:// )

4. Fixing it locally

This was quite easy, just allow insecure connections in chrome by adding extra args

"chromeOptions":{"args":["--disable-gpu","--headless", "--disable-web-security"]}}

5. Fixing this in drupalci

This was tricky as there is no clear explanation or doc on that. After debugging code I've found it is actually doable by taking over getMinkDriverArgs() method, so I've added this:

protected function getMinkDriverArgs() {
  // drupalCI chrome is executed via http://
  // for example: http://chromedriver-jenkins-drupal-contrib-652354:9515
  // due to this, we hit cross-origin errors when fetching ext. resources.
  $args = json_decode(parent::getMinkDriverArgs(), TRUE);
  $args[1]['chromeOptions']['args'][] = '--disable-web-security';
  return json_encode($args, JSON_UNESCAPED_SLASHES);
}

Modifying only args part, as we don't want to touch other parameters ( f.e. chrome URL )

Now the tests are passing again, I've cleaned up the MR a little and merge it into the dev branch

Wow, worth a blog post, and addition of the solution to Running PHPUnit Javascript tests → .

Definitely!
I've just added new section in the docs, so the info won't get lost ( couldn't find any useful info in documentation before )
see https://www.drupal.org/docs/automated-testing/phpunit-in-drupal/running-... →

Automatically closed - issue fixed for 2 weeks with no activity.