URLs with special characters like diacritical marks are counted as broken links

Problem/Motivation

If Entity Mesh founds a link like this:

https://mysite.com/url-with-diacritical-mark-ó

It displays it as:

https://mysite.com/url-with-diacritical-mark-ó

And it thinks the link is broken because the URL return 404.

Although it is not recommended to have such URLs, they are still valid if they are url encoded (so ó is turned into %C3%B3). Browsers admit typing URLs with chars like ó (although I guess under the hood they url encode them before doing the actual request).

Tested with 1.6.1 but I guess 2.x is also impacted.

Steps to reproduce

Create a content that points to an URL with special characters like óíáéí, for example. One way is to have a file field with transliteration disabled and upload a file with this characters in its filename.

Proposed resolution

It seems at some point Xray processes the links somehow, converting the special chars in other chars. We should change this, probably url-encoding the links before the processes that alters the char.

Remaining tasks

Find the root cause.

User interface changes

None.

API changes

None.

Data model changes

None.