Routing system + authentication system + format-specific routes (e.g. those in rest.module) = frustrating, unhelpful 403 responses instead of 406 responses

Problem/Motivation

While working on #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method → , I encountered 403 responses when I expected to get 406 responses. Let me quote #2737719-38: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method → , which is where this is being reported:

The struggle is because of the weird interaction/design of the routing + authentication system, which causes things to fail miserably in an unfriendly way when the user forgets to specify ?_format=nonsense, or has a typo (for example ?_format=haljson on a URL that already has a HTML route.

It's very very very confusing how Symfony's routing system first grants access because a user is authenticated via basic auth, and so AccessAwareRouter says everything is okay, and then afterwards, during AuthenticationSubscriber::onKenrelRequestFilterProvider(), we choose to deny access after all because basic_auth isn't a globally allowed access provider, and the matched route (which matches the HTML, non-REST route due to the missing ?format=) is lacking an _auth route option. You can either get a 406 (which is what we want), or a 403 with HTML (in case of non-specified or invalid format) or a 403 in the expected format (in case of non-specified or invalid format plus Accept header, even though we don't actually support Accept headers… thanks to DefaultExceptionSubscriber calling Request::getAcceptableContentTypes()).
Worse, it's even possible to get the appropriate 406 response in case of an invalid format, but for that response to be sent with Content-Type: text/html! (In case of anonymous.)

More edge cases than I can count.

Seems like the authentication system and routing system are integrated in a brittle, confusing, backward manner. And the content negotiation system makes it that tiny bit extra unpredictable. This is no one's fault. This is complex software with (too) many layers, and without comprehensive integration tests, this sort of thing is to be expected.

Proposed resolution

TBD

Remaining tasks

1. Determine root cause.
2. Propose possible solutions
3. TBD

User interface changes

None.

API changes

TBD, hopefully none.

Data model changes

TBD, hopefully none.