Entity reference translates to default language instead of source language when inline entity form is not open

I have a content type that has an entity reference to a custom entity. When the content is translated the custom entity isn't translated to the source language, but is translated to the default language.

If the inline entity form is open it translates correctly.

1) Create a US - English Node
2) Translate to English, British, change content to have a "- UK" on all field to see the change
3 Translate content to (English, Singapore) with (English, British) as the source language (with IEF closed)

- Accordion 2 Column is a custom entity. The following image is with the IEF closed. When Save is clicked the custom entity has the original language values (English)

The following image is with the IEF open. When Save is clicked the custom entity has the correct translated language values (British, English)

The problem here, I have a bunch of sections and having the content admin have to open all these sections just to get the correct language isn't ideal.