File-based media entity source fields work fine with the current solution of media_entity_download.
I guess such source field type where the main use case for the creation of this module.
Though media_entity_download becomes much less useful or not applicable at all for other media entity source field types where - for example - the actual media asset is provided from some remote site and only thumbnails are stored as Drupal managed files.
Therefore I suggest to create download handlers per media source field type. Such may be plugins and leveraged by DownloadController on demand.
Having field type specific handler may also open tracks to integrate other aspects based on the media source. I'm thinking of access control, special HTTP responses and response headers, processing of source assets before delivery (like image styles derivatives), ...
Needs work
2.0
Code
Not all content is available!
It's likely this issue predates Contrib.social: some issue and comment data are missing.
I do like Coops solution as it is very modular and flexible!
But based on Berdir's request to keep it lightweight I have created an alternative implementation with a simple hook in issue #3351543 ✨ Introduce hook to alter the download response Active .