Custom Display Settings - Allow custom ordering of view modes

Problem/Motivation

In Manage Display / Custom Display Settings tab (e.g. admin/structure/types/manage/article/display) the view modes are ordered based on the display name.

It'd be great to allow these to be custom sortable, as in my case I want to have a non-alpha order, ideally setting the order with 'draggies' at /admin/structure/display-modes/view

Eg; my views modes are: 'Inline, Small, Medium, Large, Extra large' , but the alphabetical order gives them 'Extra large, Inline, Large, Medium, Small'.

Any suggestions to accommodate this??

(Uplifted from @eddmonkey's comment on https://www.drupal.org/project/drupal/issues/2858569 → )