Access check on book pages not working

Problem/Motivation

Hi, I'm using Permission by Term on a project and it works fine. Except on book pages.
Can not figure out why it isn't working.

Steps to reproduce

Using Drupal Core 9.3.5 with core book module and Permissions by Term 3.1.17.
Created a taxonomy vocabulary 'Access groups' and some terms.
On creating terms I set up access right and selected a role to grant access to nodes via this term.

I have tested with 'Access printer friendly version' on book permissions on and off, but no changes.

Anyone experiencing the same issue on book pages?
Anyone an idea on how to fix this?

Thanks in advance.