Memory Error with D7 Migration

Created on 26 February 2024, 10 months ago
Updated 12 July 2024, 5 months ago

Problem/Motivation

PHP Fatal error: Allowed memory size of XXXX bytes exhausted.
With heavily used drupal 7 privatemsg database migration exceeded memory limits.

Steps to reproduce

Run drush pmsg1to2 with a large dataset in the pm_message_old and pm_index_old.
Issue found with 300k+ messages.

Proposed resolution

Limit data query to pull only half of the data instead of using PHP if statement to filter data out of larger query.

🐛 Bug report
Status

Fixed

Version

2.0

Component

Code

Created by

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @joshua-jake
  • Merge request !12Update PrivatemsgCommands.php → (Closed) created by joshua-jake
  • Pipeline finished with Success
    10 months ago
    Total: 262s
    #104444
  • I found that the pm_message_old table did not have an index on `mid` and that was slowing down the migrations as well. I wasn't sure the best way to add that to code because it is simply imported/old data and not part of the schema. Adding the primary key to my own table manually drastically cut down on migration time and may help others as well.

  • ivnish Kazakhstan

    Hi @joshua-jake!

    What version of D7 privatemsg do you use? I have 90% ready of migration from 7.x-1.4 version. You can test it soon, if you want.

  • @ivnish,

    Currently running 7.x-1.4 and prepping for a migration. I did not realize you were working on adding a migration process for privatemsg. I would be happy to test it.

  • ivnish Kazakhstan

    try it now :)

    1) Use dev-version of privatemsg

    2) Migration from Drupal 7 (7.x-1.4) to 2.x howto 📌 Migration or upgrade path Active

  • Status changed to Postponed: needs info 8 months ago
  • Sorry for the delay on testing. I was able to fully test the D7 migration in dev and it seems to work quite well. Only a few minor tweaks needed made to the yml files for mine to succeed. I'm not certain if these are universal changes or just for my specific migration. The required dependencies on my setup are all prefaced with upgrade_

    For example my privatemsg_thread.yml migration_dependecies section looks like this:

    required:
        - upgrade_d7_user
        - upgrade_d7_privatemsg_email_notify
        - upgrade_d7_privatemsg_message
    

    So I changed all the ids and of the migration files themselves and the reference dependencies and it worked without needing any other changes. It was quite a large and varied set of data that I had to migrate as well, so well done and thank you.

  • Status changed to Fixed 7 months ago
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024