If the entity reference field is a user reference, it should add a further condition: "Is user" (or sth. like that).
This allows to define View / Edit / Delete access, if the user is referenced directly. That makes sense, as using the "View" (referenced user account) wouldn't make much sense in most cases.
Fixed
1.0
Code