I don't think this was being done before, but might have missed it.
I've made a few settings plugins for my own use cases, things like 'sticky to top' and 'hide exposed form' etc, having the plugin interface is really good.
I found adding the parent field to the view very handy. From there you can get parent entity, which in my case is a paragraph, and therefore all the other sibling fields. This data/context can be useful.
I added :
'field' => $item,
to
$view->element['#viewsreference'] = [
'field' => $item,
'data' => $data,
'enabled_settings' => array_filter($this->getFieldSetting('enabled_settings')),
];
in viewsreference/src/Plugin/Field/FieldFormatter/ViewsReferenceFieldFormatter.php
This allows me to access the parent field and entity in my plugin's alterView
function using :
$parent = $view->element['#viewsreference']['field']->getEntity();
I've attached a patch that I'm applying to the "2.x-dev" version in composer.
Might be useful to someone else.