- Issue created by @Spokje
- 🇬🇧United Kingdom catch
Anecdotal information:
These only every happen on full test runs. I can't reproduce them locally, and they don't seem to affect the gitlab performance test job which runs tests one at a time. If they did, then we'd be able to see the extra queries in the grafana dashboard which would make them easier to track down.
This suggests it is probably a timing thing that happens when the server is under load. Considering there are image derivatives and asset aggregrates that get generated in these tests there is an inherent possibility of timing differences.
One possibility might be to take these out of the regular test runs, and instead run the performance test job on MRs/commit/scheduled runs too so that the tests always run one at a time (i.e. exclude @group OpenTelemetry from functional js jobs, would mean doing something with the standard performance tests then, but I've already been considering making those more like the Umami tests and adding them to the dashboard).
The main issue with that is the job takes about 8 minutes to complete so it would singlehandedly drag upwards total pipeline times. When I tried to consolidate test methods it caused trouble with chromedriver log parsing, but might be time to try that again.