large site, migrate import is very slow

Hi hbaa,

First of all, thank you for your interest in the migrate wizard module.

The module simply generates the yml files for the migrations that will later be used by the drupal (core) migrate module.

If your portal contains so many nodes, it will inevitably take a long time to migrate.

Tips to make it take less time:

Use a machine with more resources, I often program and test the migrations in my local environment of the projects I work on, but then when launching the migrations in the pre-production environments, they run faster because I have machines with more resources.

File migration with migrate wizard has some peculiarities:
the files are copied one by one on the fly when migrating the nodes, you do not need to migrate the files or media first, although if you do, this will be scalable and each entity will be reused when appropriate so that downloaded files or entities pointing to the same file are not repeated.
In any case, to speed up this part, I recommend placing the files in the destination folder, so that in the migration process, each file will not be downloaded and the file or media entities will simply be created when appropriate.
This is done this way because in some cases you may not want to migrate all the files that exist in the source; when migrating the nodes, only the files that are used in the migrated content will be copied.
That is, if you have 20,000 files in the source but you only want to migrate those that are being used in a type of content, do not migrate the files, only migrate the type of content you want and only the files of that type of content will be copied, for example.

Regarding the errors you mention, I would need more data from the trace, to see which part of my module it comes from, although from what I see in the error, from afar, it may be due to your version of php, or the data you are trying to import.

In any case, please investigate the error and if you see that it is due to the module, open a specific issue with it.

I'm closing the issue, best regards and good luck with your migration.

If you are using DDEV with "docker desktop" on macos, go to settings>resources and set the cpu limit to the maximum. You might want to up some of the other resources too.

There is also a replacement for docker desktop called OrbStack, that "should" run it faster, although I haven't tried it yet.