routeHasModal return value breaks form alter check

Problem/Motivation

The change of the return value of routeHasModal in #3324803: Add optional parameters for indiviual routes → breaks the check in route_in_modal_form_alter(), causing all routes without parameters in the module config to be regarded as route without modal.

Proposed resolution

Fix for now would be to check if the return value of routeHasModal equals FALSE.

Better would be to also revert the return value of routeHasModal() to boolean only, and provide new function to get the route parameters.
This is something for a separate issues though.