Formatter third party settings missing from Views UI

Moving this to 10.1. It would be amazing if we could get some momentum going on this issue again.

Attached a patch which provides that support for fields.

It should work in a way that ensures that the third_party_settings is entirely optional, keeping the existing views behaviour where possible.

With the new schema change think we will need an upgrade path + tests.

Patch works, but I find it weird UX to have to ticky a box that says 'Use third party setting', especially when the third party settings form is hidden in a details element anyway.
It would be pretty much the same to just show the details element and have it closed by default.

Though I think it would be better UX to have the formatter settings AND the third party settings together in a detail element, so it looks the same as in entity display settings. Although that change would be out of scope of this issue.

Tagging as a contrib project blocker, because this affects any contrib module that offers extra options for field formatters.

Actually, the patch doesn't completely work -- I can't remove settings from a field after saving the view:

1. Edit a view, set a third party setting
2. Save it
3. Remove the setting - unselect both the setting AND the 'Use third party setting' checkbox
4. click 'Apply'
5. The settings are still used in the preview output, and editing the field again shows they are back

I think there's some confusion between

$options['third_party_settings']

and

$options['settings']['third_party_settings']

The field formatter system is going to expect third_party_settings to be within its settings, so I think putting them outside of that in the Views config is a mistake, as it just confuses things.