Batch processing doesn't really work.

Created on 9 April 2022, over 2 years ago
Updated 9 June 2023, over 1 year ago

Problem/Motivation

Not sure if I am stating this correctly and yes, this is version 4.0.0 as havent migrated to D9 yet (next week hopefully) - so maybe this has been fixed?

So yes, it does batch process the updates but still easily runs out of memory before getting to that point based on the total number of items. Is it not possible to "fully batch" this process?

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

πŸ› Bug report
Status

Closed: cannot reproduce

Version

4.2

Component

Core

Created by

πŸ‡¨πŸ‡¦Canada liquidcms

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡¨πŸ‡΄Colombia jedihe

    Running VBO 4.2.3, I can confirm batching does work: I was hitting PHP's max memory with a batch size of 20, then I started tweaking the batch size to a lower value until processing ran just fine, all items were correctly processed.

    This report may be related to #3227671: VBO batch process seems to be broken after #3167619 β†’ .

  • πŸ‡ΊπŸ‡ΈUnited States aaronelborg

    I'm not sure if this is the best place to ask this so forgive me if I'm out of place here...

    I'm wondering if OP's issue stems from the fact that they're accessing properties within the execute() function from a loaded single entity but performing operations on many items that were referenced by said single entity. My assumption is that the bulk operation is only aware of that single entity. I'm actually running into this issue now and a search found this issue so here I am.

    In short, I'd like to get my custom VBO action to access bulk operations messaging/saving operations... but not on the $entity passed to execute. Instead, I need this data to reference the entities referenced from the single $entity passed to execute().

    For example, if I'm loading nodes tagged with a certain term and then removing that term from these nodes, how can I access the bulk operations data to do its operations (messaging, count of items, etc) on these nodes? Obviously, they aren't the $entity passed to the execute method so (I'm assuming) that the bulk operations has no knowledge of their existence. What I'm using works but it will time-out occasionally on terms that reference tons of nodes. As it stands, I just re-run it until I actually see the progress bar finish.

    I've found very little documentation on this which leads me to believe I might be doing something weird and/or wrong. Thanks for any advice.

          $nodes = \Drupal::entityTypeManager()->getStorage('node')->loadByProperties([
            $old_vocab_field => $tid,
          ]);      
          foreach ($nodes as $node) {
            // Add tid to node.
            $node->set($new_vocab_field, $tid);
            foreach ($node->get($old_vocab_field) as $k => $v) {
              $target_id = $v->get('entity')->getTargetIdentifier();
              if ($target_id == $tid) {
                // Remove old tid from node.
                $node->$old_vocab_field->removeItem($k);
                $node->save();
                break;
              }
            }
          }
  • πŸ‡΅πŸ‡±Poland Graber

    @AaronELBorg I think you need to use batch API for your processing and not use VBO but a custom solution. Batch in a batch will not work I think..

  • πŸ‡΅πŸ‡±Poland Graber

    Or differently: create e view that displays what you actually need to process.

  • πŸ‡ΊπŸ‡ΈUnited States aaronelborg

    @Graber, Yeah as soon as I hit the 'submit' button on my post, I thought "yeah... this could be done by accessing the node ($entity) directly instead". I appreciate your input though. Thanks!

Production build 0.71.5 2024