Set formatter settings at media entity level ?

This isn't really supported by the module. The only idea that comes to mind to achieve this would be to implement your own formatter plugin, that would pass to the template height/width that would come from the media entity field values (which you would create in your project), instead of from the formatter config.

As @marcoscano mentioned, the ability to set the formatter settings at media entity level isn't really supported by the module so closing this issue. The suggested solution is to implement your own formatter plugin, that would pass to the template height/width that would come from the media entity field values.