I had the same issue. You have to create a preprocessor or tweak the content_access one. If i'm understanding this correctly : Search Api takes all the grants and if one is ok then the page is shown. I've resolved this by changing the addFieldValues function in contentAccess.php. Now, it first check if there is a grant in the permissions_by_term realm. If so it only take those grants. If not it works as normal.
It seems that the function to check if the media should be checked use the taxonomy configured in the permission by term module but then when the permission is really checked, that taxonomy is not used. The access checker just check if the user has a right on one of the taxonomy field.