Add ui_styles_manage_display sub module

Should it be ui_styles_manage_display or ui_styles_field_formatter?

You are right Grimrepaer, let's stop thinking about config entities and their backoffice UI but about plugins.

https://www.drupal.org/project/fences → allows to control wrappers tag and/or remove wrappers. It also allows to inject classes. Maybe working with this module will do the trick.