How to restrict access to node

Problem to restrict access to node with multiple parents

Hello, I have some problem with restriction of group node access.
I have 2 group types, Group A and Group B for example, both has installed content Group node (Basic page), in Group A basic page is public and visible for all kind of users. In Group B basic page is accessible only for group members with "Access Node" group role. When node has only Group B parent, then all is working, node is accessible only for group members with "Access Node" group role. But when node also has Group A parent it is becoming public even if Group B is one of parents. I did a lot debug in group_entity_access hook here is the debug example

array:1 [▼
  0 => "group_node:page"
]
Drupal\Core\Access\AccessResultAllowed {#2964 ▼
  #cacheContexts: array:1 [▶]
  #cacheTags: array:13 [▶]
  #cacheMaxAge: -1
}
array:1 [▼
  0 => "group_node:page"
]
Drupal\Core\Access\AccessResultForbidden {#13372 ▼
  #cacheContexts: array:2 [▶]
  #cacheTags: array:13 [▶]
  #cacheMaxAge: -1
  #reason: null
}
"second time only it is returning isForbidden"
array:1 [▼
  0 => "group_node:page"
]
Drupal\Core\Access\AccessResultForbidden {#18119 ▼
  #cacheContexts: array:2 [▶]
  #cacheTags: array:13 [▶]
  #cacheMaxAge: -1
  #reason: null
}
"second time only it is returning isForbidden"
array:1 [▼
  0 => "group_node:page"
]
Drupal\Core\Access\AccessResultNeutral {#13212 ▼
  #cacheContexts: []
  #cacheTags: array:12 [▶]
  #cacheMaxAge: -1
  #reason: null
}

Question is how to give some weight to Group B permissions or how to return $access::forbidden() if there is one restriction.