Assign task to member in group

I am trying to create a node in a group and assign it to a student. I have a node type with a referens field. I can then choose any user from the site from the list, but I want the group admins only to see the user in the group.

I have a view to the field but the contextual filter does not seem to work. Its all or none.

Can this be done?

Thanks for any input!
/Hans