When scraping our site, Google Adsbot is receiving 406 responses from Drupal. This seems to be triggered by the directives supplied in the Accept header of the request:
application/xml;q=0.9,text/html,*/*;q=0.8,application/xhtml+xml
The Services module is apparently using the Accept header for content negotiation in the following class method:
\Drupal\services\StackMiddleware\FormatSetter::handle()
From what i can see, this method is mishandling the Accept header, and setting the request format to the first value it finds in the Accept header - XML in this case – which Drupal is unable to serve. However, in actual fact, HTML and XHTML are perfectly acceptable formats according to the header.
Probably what the code should be doing is parsing all the directives supplied in the Accept header, sorting by q-factor (i.e. weighting), and then setting the request format to the best available option?
Thanks
Active
5.0
Code
Not all content is available!
It's likely this issue predates Contrib.social: some issue and comment data are missing.
Looks like the related code did not change in 5.x yet.
Perhaps one of you could prepare a MR to fix this, if you're still affected or sponsor a developer to do so?
This is open source and the maintainers seem inactive or busy. Would be great if the community could help.
As a first step: How can this be reproduced? Does this only happen on service definition URLs or on all URLs of a project using this module?