Access Caching seems wrong in some contexts

Problem/Motivation

Hello,
I encountered a very strange issue where the action "Masquerade as XXXX" link was missing from the dropdown on some users (in a view). I tried to understand what was wrong but couldn't find what was wrong exactly.

The issue seems related to cache though because all links show up after flushing all cache.

I tried to analyze the code and everything seem to use "session.is_masquerading" as a cache context. I wonder if that context is sufficient in all situations...
An Access Denied is most likely cached, and that's why the link is not showing up.
I went through the code, tried to print stuff but it is useless because since the problem is cached, it doesn't even go through the code. The hard part is to find out the way that access result has been cached wrong.

Also tried to use two different browers to login in both and use masquerade at the same time to see if it was creating any conflict, but no success since I noticed that the parameter bag seem to store only one ID.

Another possible origin of the issue, I noticed that my site is using peristent_login. Maybe there is some incompatibility between the two modules?

Steps to reproduce

I couldn't figure out how that happened. I just have a database with the bug cached, but couldn't reproduce it again after flushing cache.

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes