Option to override batch size

The new option might take some time to be fully implement. A "security" max limit could be useful in the meantime.

Assigning myself to review & test the PR.

The tests are passing on my local however they are failing in CI.
Can someone share some wisdom on it?

lando phpunit --group=migrate_tools
PHPUnit 8.5.31 by Sebastian Bergmann and contributors.

Testing
............................

......
.. 36 / 36 (100%)

Time: 4.84 minutes, Memory: 353.00 MB

OK (36 tests, 328 assertions)

HTML output was generated
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...
https://drupal-contributions.lndo.site/sites/simpletest/browser_output/D...

PR - https://git.drupalcode.org/project/migrate_tools/-/merge_requests/19
Looks good to me.

This patch contains the current state of the merge request. See "Patches from drupal.org merge request URLs are dangerous?" for more information.

The latest patch provided here didn't apply, so I created a new one from the latest diff of merge request 19.

Updated the patch with a small fix to the option name which was incorrect, causing the batch size to not work (thanks to olli → for spotting it).

Hoping if things are all good here that we could get this into a future release for the module? This would be beneficial for some people we're working with and would love to see it included :)

Much like the above comment, is there a date of release for this module please? Thanks team!

forgot to unassign from myself

attached is the patched re-rolled against 6.0.x

I'm ignoring the patches (though I understand why some people are adding them).

We will use the MR. Since there are 2 open. we will use the more recent reroll.

But, the latest MR (19) needs a merge conflict resolved, so moving to needs work:

https://git.drupalcode.org/project/migrate_tools/-/merge_requests/19/con...

Back to needs review.

I've compared MRs 7 and 19 to see what has changed other than coding standards and formatting and these are the things I found.

-    $migration_dependencies = $migration_plugin->getMigrationDependencies();
+    $migration_dependencies = $migration_plugin->getMigrationDependencies(TRUE);

-      $migration_dependencies = $migration->getMigrationDependencies();
+      $migration_dependencies = $migration->getMigrationDependencies(TRUE);

+   * @option batch-size Optionally use batch iterations, with a limit on the
+   *   number of items to process in each batch iteration.

The description is substantially shorter in the latest MR, but then that makes it much easier to read. Perhaps the extra details could be moved into the README or some other documentation.

+      // Integer cast required because MigrateBatchExecutable requires the
+      // `update` and `force` options to strictly be integers.
+      $options['update'] = (int) $options['update'];
+      $options['force'] = (int) $options['force'];

+    // If this batch is run via Drush, we need to initialize the progress bar
+    // for the background process.
+    if (PHP_SAPI === 'cli') {
+      $output = Drush::output();
+      $output->setDecorated(TRUE);
+      // Initialize the Symfony Console progress bar.
+      \Drupal::service('migrate_tools.migration_drush_command_progress')->initializeProgress(
+        $output,
+        $migration,
+        $options,
+      );
+    }
+    
+    $executable = new MigrateBatchExecutable($migration, $message, $options);

   public function onPostRowDelete(MigrateRowDeleteEvent $event) {
     if ($this->feedback && ($this->deleteCounter) && $this->deleteCounter % $this->feedback == 0) {
-      $this->rollbackMessage(FALSE);
+      $this->progressMessageEmit(FALSE);

I got confirmation from @heddn via Slack that these changes above look okay so this can move forward with a round of testing for the *new* MR. Once that happens and this is RTBC with the new MR, then we can see if @heddn or one of the other comaintainers can get this in.

Sucessfully tested MR 19 on a large custom migration (42000 nodes) with --batch-size of 1000 and 2000, with and without the --update flag, in a local environment and on a dev server. migrate-import correctly processes the given amount of items, and the duration for each batch stays similar.

Hmm... that's disappointing. Anyone else able to test?

What's disappointing? emixaam's comment seems ideal, no?

I found that the `--force` command does not mix well with `--batch-size` either... it fails all migrations if their dependencies are not fulfilled. If I run a migration without --batch-size using --force, it works as expected, ignoring the dependencies.

I must have misread it before so please ignore my previous comment.

Besides the Array to String conversion, this patch also is unable to accept custom drush migrate:import parameters:

AWS:[wwwdev@mkt-qal-drupal01 src]$ drush mim --tag=node_translation --continue-on-failure --language=es,it --batch-size=1000
[warning] Array to string conversion Process.php:341
> [error] Migration failed with source plugin exception: The "language" option does not exist. in /data/3ds/swd10/releases/4.19.12-RC/src/vendor/symfony/console/Input/Input.php line 140

I assume this will also affect the --migrate-debug flag.

These are both added using official drush extension setups:

services:
  sw_migrate.commands:
    class: \Drupal\sw_migrate\Commands\SwMigrateCommands
    tags:
      - { name: drush.command }

<?php

namespace Drupal\sw_migrate\Commands;

use Consolidation\AnnotatedCommand\CommandData;
use Drush\Commands\DrushCommands;

/**
 * Extends migrate-import with a language-filtering option.
 */
class SwMigrateCommands extends DrushCommands {

  /**
   * Register new options for the migrate:import command.
   *
   * @hook command migrate:import
   * @option language Enable language filtering
   * @usage drush migrate-import --language=fr,it
   *   Run translation-based migration(s) in the specified languages only.
   */
  public function additionalOptionsMigrateImport(CommandData $commandData, $options = ['language' => NULL]) {
    // No action required here. The new options will be examined during the
    // source plugin if required.
  }

}

... so I am surprised that adding the --batch-size parameter is suddenly ignoring them?