batch_size creates an infinite loop

Created on 14 March 2023, over 1 year ago

Problem/Motivation

When using batch_size in the source config, it creates an infinite loop.
It seems to import everything in a first batch then loops infinitely on batches containing the whole dataset.

Steps to reproduce

Here is an example migration that can reproduce the problem :

id: folio_migrate_pays
label: Pays
migration_group: folio
source:
  plugin: custom_sql_query
  key: folio_catalogue
  keys:
    - id
  track_changes: true
  batch_size: 2
  sql_query:
    #language=mysql
    'SELECT id, name FROM v_countries'
destination:
  plugin: 'entity:taxonomy_term'
  default_bundle: pays
process:
  name: name
  field_identifiant_catalogue: id
dependencies:
  enforced:
    module:
      - folio_migrate

If I call drush migrate:import folio_migrate_pays --update, it hangs forever.
If I look at what it is doing, it seems it is calling CustomSQLQuery::fetchNextBatch() again and again forever.

Proposed resolution

In the code, it looks like $batchSize is never really used.

SqlBase uses it to add a range to the query. However, in our case I am not sure that's possible, since the query is not manipulated in PHP.

🐛 Bug report
Status

Active

Version

1.3

Component

Code

Created by

🇫🇷France prudloff Lille

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024