MigrateBatchExecutable rewinds the iterator each batch iteration

Created on 16 December 2022, over 1 year ago
Updated 13 November 2023, 8 months ago

Problem/Motivation

If attempting to process a large source via the MigrateBatchExecutable the iterator rewinds the source each pass. This can have performance implications given a large source or a source plugin that has large upfront loading tasks.

Steps to reproduce

Can follow the comment in https://www.drupal.org/project/migrate_tools/issues/3253741#comment-1482... β†’ .

Proposed resolution

Re-write the MigrateBatchExecutable to not leverage core's import method which expects to be executed once in a singular thread: https://git.drupalcode.org/project/migrate_tools/-/blob/0813c0fb91406695....

The batch could instead be structured to have an initial operation responsible for enqueueing rows into a queue to be processed from the source (singular read). The follow up operation is then response for claiming item(s) (rows) and doing the actual migration processing.

πŸ› Bug report
Status

Active

Version

6.0

Component

User interface

Created by

πŸ‡¨πŸ‡¦Canada JordanDukart PEI

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Production build 0.69.0 2024