I've enabled the permissions_by_entity submodule.
Added a taxonomy reference to the media bundles
User permissions are to edit, create, delete media
Users seem to be able all media regardless of their term.
Active
3.1
Code
Not all content is available!
It's likely this issue predates Contrib.social: some issue and comment data are missing.
It seems that the function to check if the media should be checked use the taxonomy configured in the permission by term module but then when the permission is really checked, that taxonomy is not used. The access checker just check if the user has a right on one of the taxonomy field.