Formatter third party settings missing from Views UI

Moving this to 10.1. It would be amazing if we could get some momentum going on this issue again.

Attached a patch which provides that support for fields.

It should work in a way that ensures that the third_party_settings is entirely optional, keeping the existing views behaviour where possible.

With the new schema change think we will need an upgrade path + tests.

Patch works, but I find it weird UX to have to ticky a box that says 'Use third party setting', especially when the third party settings form is hidden in a details element anyway.
It would be pretty much the same to just show the details element and have it closed by default.

Though I think it would be better UX to have the formatter settings AND the third party settings together in a detail element, so it looks the same as in entity display settings. Although that change would be out of scope of this issue.

Tagging as a contrib project blocker, because this affects any contrib module that offers extra options for field formatters.

Actually, the patch doesn't completely work -- I can't remove settings from a field after saving the view:

1. Edit a view, set a third party setting
2. Save it
3. Remove the setting - unselect both the setting AND the 'Use third party setting' checkbox
4. click 'Apply'
5. The settings are still used in the preview output, and editing the field again shows they are back

I think there's some confusion between

$options['third_party_settings']

and

$options['settings']['third_party_settings']

The field formatter system is going to expect third_party_settings to be within its settings, so I think putting them outside of that in the Views config is a mistake, as it just confuses things.

Thanks for working on this! The MR works really well and fixes missing Captions settings in PhotoSwipe in Views.

About the current patch and removing settings, I can only partially replicate it. If I set a value like 'Image alt tag', collapse it by disabling 'Use third party setting' checkbox, and save, the setting does remain, like @joachim commented in #31.

But if I instead first set it to "- None -" and then save, it's no longer active, regardless whether I collapse it or not.

I do think it should be totally disabled (set to "-None -") if the setting is collapsed, by unchecking the 'Use third party setting' checkbox.

Since this can be very hard to debug and understand, and options are mysteriously missing from the Views UI in Contrib modules integration without any clue, resulting in a more or less "dead end", could this issue be considered bumped up to "Major"?