Anonymous simple inline entity form permission issue

Problem/Motivation

I have a content type with a simple inline entity form field on it. Then I go to add new content and I use a form_alter on that parent content to add a previously created entity reference to the inline entity form. We basically do the following in the form_alter to add this default value.
$form['field_xxxxxx']['widget'][0]['inline_entity_form']['#default_value'] = $default_entity;
This is working for authenticated user but not for anonymous.
So I have to turn on 'Anonymous users can create xxxxx content type' in the permissions. So I basically have to give anonymous users create permission even though I have already created and saved the referenced entity. Is that a bug or is there something else I should set in the form alter when I set this default value to let inline_entity_form know that this is not a 'create' any longer? Like maybe I have to change the #op value from 'add' to 'edit' or something? Thanks for any help.