Recursive loading of relationships causes memory errors on save

First pass at a patch to add the config option

Re-roll to work with patch in https://www.drupal.org/project/gatsby/issues/3356434 🐛 Large number of log entries leads to memory/timeout issues Fixed

I've tested out this patch in hopes that it fixed the issue I'm encountering, but it does not. My issue is saving an Entityqueue in Draft causes a memory error:

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /code/web/core/modules/jsonapi/src/Normalizer/NormalizerBase.php on line 43

I'll keep trying with this issue, but I may spin up a new one if it doesn't look to be related.

There is this setting: Do not store data for entity types that are self referenced available. I don't know if you all have tried that or not. Memory issues sound related to this as it could point to an infinite loop of relationships possibly.

I do have that checked @apmsooner. I've recommended to the team that we don't version entity queues for the short term. I'll create another issue to look into it.

That setting doesn't help us, as the directly related entities aren't the same type/bundle, and this common scenario for us:

• Have a Node related to multiple Storage entities.
• Have those Storage entities related to multiple Nodes, of the same or different bundle, as they are reusable components.
• Have a need to trigger Gatsby builds when Storage entities are saved, as content editors will also edit Storage entities directly (not just through Node edit forms).
• Use Inline Entity Forms on the node bundle, so that content editors can add/edit Storage entities directly on the Node edit form.

This means when a Storage is edited on the Node form, and the node is saved:

1. The Node save triggers Gatsby, which recurses through all the relationships of the Node.
2. Inline Entity Forms triggers a save on the edited Storage entity, which triggers Gatsby, recursing through all the relationships of that Storage entity.
3. With the option Do not store data for entity types that are self referenced checked:
   • #1 does not recurse through Nodes of the same bundle. It does recurse through other relationships of the related Storage entities, which may include Nodes of a different bundle - and all of their relationships.
   • #2 is a Storage entity that trigged Gatsby, so it gathers all of the relationships of that Storage entity - which will include the original Node, and not exclude any other Nodes of that bundle, so there may be many. And each of those nodes may be related to 0 or more additional Storage entities, of a variety of bundles. Which may also be related to other Nodes, of the same bundle as the original.

We do not need the recursive relationships in this scenario. If we do not recurse through them, the Node that was edited will have one level of relationship, which will include the Storage that was edited. The Storage that was edited and saved at the same time will have one level of relationship, which will include all other Nodes that reference this Storage. That's all we need updated in the fastbuild, and does not cause memory errors.

Enabling this patch has brought some insight into a @todo comment in GatsbyEntityLogger.php.

// @todo Document why this is necessary for this entity type but not others.
    if ($action !== 'delete') {
      if ($entity->getEntityTypeId() === 'paragraph' && $this->config->get('recursive_relationships')) {
        return;
      } ...

With this patched option enabled, paragraphs are completely siloed from being included in fastbuilds. I believe the assumption was they'd be captured in recursive crawl of relationships of parent nodes. Which makes sense to reduce overall gatsby logs.

I think this check should be enhanced a bit to at least check the new config option is set, and maybe also that paragraphs are explicitly set as an entity to pass to build/previews. Updating that legacy @todo comment would be good too.

Patch to add $this->config->get('recursive_relationships' for

if ($entity->getEntityTypeId() === 'paragraph' && $this->config->get('recursive_relationships')) {

per comment #11 🐛 Recursive loading of relationships causes memory errors on save Active

Thanks for this work! We were seeing some PHP errors on occasion:
TypeError: Drupal\gatsby\GatsbyEntityLogger::getJson(): Argument #3 ($recursive) must be of type bool, null given, called in /var/www/html/web/modules/contrib/gatsby/src/GatsbyEntityLogger.php on line 82 in Drupal\gatsby\GatsbyEntityLogger->getJson() (line 247 of /var/www/html/web/modules/contrib/gatsby/src/GatsbyEntityLogger.php).

Added a test to resolve that here.