Field visibility access is not applied correctly due to caching

Test only patch to demonstrate the issue.

I've pushed some commits to check if the issue is solved by this patch, but it seems to bring up new issues, I'll look more into it

Even if I set ->setCacheMaxAge(0); the behaviour persists, it seems like Drupal is not checking the observers access before its rendering the page, or before its caching the entity or view mode

I've reverted adding the patch as it does not seem to help, and instead added caching the account per user. Lets see the result of this.

I tested the MR on our project where the issue is reproducible, and after applying the changes the caching issue is fixed.

Thank you all, I've merged