Contrib.social

Webform submission import: failed to import a lot of records executed in multiple batches

Open on Drupal.org →
Created on 16 January 2024, 11 months ago

Problem/Motivation

I cannot import a large number of records that are executed in multiple batches.

If the number of records to be imported exceeds 100, drush wfi command splits the task into several batches. When handling 2nd or subsequent batches, the process should skip over the records that have already been imported. However, WebformSubmissionExportImportImporter skips by line rather than by CSV record.

while ($index < $offset && !feof($handle)) {
  // => We should use fgetcsv() here.
  fgets($handle);
  $index++;
}

The problem and cause is similar to this:
https://www.drupal.org/project/webform/issues/3386895 🐛 Upload count is incorrect in getTotal method of WebformSubmissionExportImportImporter Fixed

Steps to reproduce

  1. Install Webform and Webform Submission Export/Import
  2. Create a webform
  3. Insert ~200 records to the form
  4. Export the submission data by drush wfx --exporter=webform_submission_export_import
  5. Import the exported files

Proposed resolution

Skip processed records by CSV record as https://www.drupal.org/project/webform/issues/3386895 🐛 Upload count is incorrect in getTotal method of WebformSubmissionExportImportImporter Fixed .

Remaining tasks

Write tests

User interface changes

N/A

API changes

N/A

Data model changes

N/A

🐛 Bug report
Status

Active

Version

6.2

Component

Code

Created by

🇯🇵Japan kensuke-imamura

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024