Non valid entities getting added to fields

I'm not sure how this was happening, but I noticed in my own implementation of this that non valid entities were getting added to some of my fields.

The way this was set up is that there was a reference A:
node type A, node type B
field_b_refs
field_a_refs

And reference B:
node type A, node type C
field_c_refs
field_a_refs

And, somehow, node type B was getting added to field_c_refs on node type A when updating field_b_refs on node type A.

This then caused a cascading reference update that ended up with all past values getting deleted from field_b_refs (as well as the corresponding values being deleted from field_a_refs on node type B).