Leftover extra field entries in entity_view_display config entity

When saving the field ui form, Drupal will save the position of each extra field in the entity_view_display config entity.
These entries can be accessed with EntityDisplayBase->getComponents();

I found that after disabling extra_field_example, the entries are still there.
See https://drupal.stackexchange.com/questions/244305/how-to-clean-up-leftov...

I guess the module needs to do some cleanup on uninstall, and maybe after a cache clear?

Or maybe this needs to be implemented in core.