The output from Drupal.org automated tests should be useful for humans

Sometimes I'm careless and submit a patch to a project that has "automatic tests" enabled. I know from past encounters that by doing so, I set my self up for a humiliating experience. But sometimes I just want to get some bug fixed or feature added – so I brace myself and do it.

I would like to share a recent such encounter. It's about this little patch: #2497785: The path for advanced help in submodule Views Slideshow: Cycle must be updated → .

The patch is not exactly rocket science. It is a one-liner replacing a quoted string with a wrong path with another quoted string with the correct path (i.e.no code change).

It failed (of course) – my patches always fail automatic testing.

The reason given for failing is:

D7 CI error

I'm guess there was a letter-shortage in the context this message was created. But there are plenty of letters in the "Console Output" context (see below). Maybe some of them can be put to better use here?

Exploring further, I click on this message. This takes me to this page → , where the reason for failure has been further abbreviated to:

CI error

So "D7" wasn't part of the problem. That's nice to know.

And there is another link: "View results on dispatcher". Clicking it leads to this page, which seems to be the home page Mr. Jenkins.

And Mr. Jenkins' page has all sorts of fancy stuff: "Status" and "Console Output" and "View Build Information" and "Parameters". I click through all of those, and even read all the letters in "Console Output" (letters galore – too many, in fact) and I am still unable to figure out why my one line patch was a total failure.

Don't get me wrong: I think automatic testing is a great idea, I really do – but only if the feedback it provides makes sense to a human. The idea behind providing feedback must be to motivate the human to use that feedback to improve his code. What the current feedback communicates to the contributor is this: “You and your patch failed – and you are too stupid to understand why.”

I know that it's a computer and that I should not take it personal, but for me, the user experience is just bad – it is humiliating to have work rejected and not being unable to understand why.

And on a less personal level: Exposing contributors to error message made up of obscure initials, and raw "console output", is probably not the best motivation.

Update 2016-02-12:
I see from #2 by drumm that the failure of this particular patch is a bug.

Fine, but fixing this bug does not fix the real problem: The output produced by the automated testing is not suitable for humans..

The output from this particular should not have been:

D7 CI error

… plus a lot of crud – including a very long and unreadable (to humans) console message. It should have been something like this message – given the (IMHO faulty) logic of the current test bot:

The testing failed because the test bot did not find any tests for your patch.

Contributors that upload patches to fix bugs and add features to Drupal should be among the most valued members of the Drupal community. This means that we should care about them and their motivation.

In my company, we don't roll out new features without at least some usability and UX testing. It may be testing involving observing real users (i.e. not members of the dev. team) using the new features, or it may be something done by non-users (e.g. heuristic evaluation) – but usability testing is mandatory before major changes to a configuration are put in production.

Not being a member of the infrastructure team, I don't know how you go about your business, but I would suggest that you review "best practices" from the industry regarding major changes to the configuration, and perhaps adopt some of them for your workflow.