Motivation
When running tests in the GUI, it is often useful to look at the screenshots (HTML code) as seen by the test, for example if a certain word should appear in the HTML and does not appear, examining the HTML code can give us a good idea why is it not appearing.
If you have a continuous integration server running something like:
php scripts/run-tests.sh --verbose --xml path/to/artifacts my_website
You will see the test results (passes) as console output for each build, and if you save path/to/artifacts for each build, you will have access to failure messages in XML format.
Currently, though, there seems to be no way to keep screen capture HTML files as artifacts. Doing so would be useful in case a continuous integration server detects a failure for a given commit: one could then have a further tool (screen capture HTML files) to diagnose the problem.
Proposed approach
An issue is currently open,
#2253999: Provide a way to keep test results and verbose messages as artifacts →
, which proposes a new unified way to output test results for continuous integration servers to use as artifacts. I have also submitted a pull request to the drush project which allows this. The latter request has been closed because it's outside the scope of Drush. The former might take a while to be developed.
For those who need this functionality soon, a simple argument --xml-verbose to scripts/run-tests.sh might suffise, for now. So, running:
php scripts/run-tests.sh --verbose --xml path/to/artifacts --xml-verbose my_website
would do two things:
(1) add the xml file lines for the HTML screen captures, in the correct order.
(2) copy the HTML screen capture files from sites/default/files/simpletest/verbose/* to the xml directory (for example path/to/artifacts)
Once we have that, we can set our continuous integration server to save, for each build, whatever is in path/to/artifacts for later use in diagnostics.