Validation does not change with changing field formatter

Problem/Motivation

URL validation doesn't change after changing the field formatter.

Steps to reproduce

Set up a media with Remote media url and a formatter other then plain text. Save the form. Check if URL validation works by trying to add a media.
Now change the field formatter, and try to update the media. The previous validation will be still in effect, even after cache rebuild.