Building a view of entities not being referenced

Hello,

I have a project where nodes of a dedicated content type reference one of all users having a specific role.
So far, it's not a problem building a view of users filtered on the required role and output that view as an Entity Reference display so it can be used in the content type entityreference field selector.

My problem is that these users should be referenced only once so the view needs to only display users not already being referenced.
The goal: when a new content of that type is created, the entityreference field only displays available users (which have been created or unreferenced before).

In the view advanced settings, a relation can be added to the view to retreive content referencing the users which works fine but I can't find how to set the filter: if the user is not referenced, there is no content found through the relation so I can't filter like empty title or empty nid because of non existing related content.

I have some ideas but I don't feel how to set them up:
1. The relation has a checkbox "require relation" which allows (if checked) to limit result to users being referenced. I need the exact opposite feature, like "exclude relation".
2. In the view filter, I need an option like "non existing" or "is not set" or "related content not found" (because "is empty" doesn't seem to work when no referencing content is found).
3. I've also tried to alter views-view-unformatted.tpl.php (as indicated in views advanced settings, theming) to get rid of rows when related content is found but that template file seems not to be used by entityreference view display.

Maybe another view setup might do it?
Thanks