Memory exhaustion creating large number of nodes

Created on 3 June 2023, about 1 year ago
Updated 4 April 2024, 3 months ago

Problem/Motivation

If a large number of nodes is created programatically then the memory may be exhausted. I encountered this trying to create a node (or paragraph) for every outbound email after a bulk mailout.

<!--break-->

Steps to reproduce

Use the following code with and without the hook_post_action module installed:

use Drupal\node\Entity\Node;

$node_storage = \Drupal::EntityTypeManager()->getStorage('node');
$nids_processed = [];

for ($i = 1; $i <= 3000; $i++) {
    $node = Node::create([
      'type' => 'post',
      'title' => "test-post",
    ]);
    $node->save();
    $nids_processed[] = $node->id();
    if (count($nids_processed) % 500 == 0) {
      echo "Memory: " . (int) (memory_get_usage(TRUE) / 1024 / 1024) . " MB\n";
      $nids_to_clear = array_slice($nids_processed, -500);
      $node_storage->resetCache($nids_to_clear);
    }
}

Without module installed:

Memory: 72 MB
Memory: 72 MB
Memory: 72 MB
Memory: 72 MB
Memory: 72 MB
Memory: 72 MB

With module installed:

Memory: 44 MB
Memory: 76 MB
Memory: 108 MB
Memory: 140 MB
Memory: 172 MB
Memory: 204 MB

Proposed resolution

A couple of possible solutions:
1. Change callback arguments.
Rather than add the entity to the callback arguments add the type and id instead. So for example on insert this function:

function hook_post_action_entity_insert(EntityInterface $entity) {
  drupal_register_shutdown_function('_hook_post_action_post_save', $entity, 'insert');
}

becomes:

function hook_post_action_entity_insert(EntityInterface $entity) {
  drupal_register_shutdown_function('_hook_post_action_post_save', $entity->getEntityType(), $entity->id(), 'insert');
}

and in the corresponding callback:

function _hook_post_action_post_save(EntityInterface $entity, $op) {

becomes:

function _hook_post_action_post_save(EntityTypeInterface $entity_type, int $entity_id, $op) {
  $entity = \Drupal::entityTypeManager()->getStorage($entity_type->id())->load($entity_id);

2. Allow/block list.
Support an allow list or block list of entity types to exclude from a post save action and check against these in the hook_post_action_entity_ functions.

✨ Feature request
Status

Active

Version

1.2

Component

Code

Created by

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.69.0 2024