I had the same issue. You have to create a preprocessor or tweak the content_access one. If i'm understanding this correctly : Search Api takes all the grants and if one is ok then the page is shown. I've resolved this by changing the addFieldValues function in contentAccess.php. Now, it first check if there is a grant in the permissions_by_term realm. If so it only take those grants. If not it works as normal.