Preserve NIDs from Drupal 7

Created on 15 October 2024, 6 months ago

Problem/Motivation

In order to avoid a massive redirect table, we want to preserve the Drupal 7 NID in the migrated content. This is not the case now.

There are currently around ~2 million nodes. The challenge is that these are live migrations, so content is both added to the Drupal 7 site (and then migrated), as well as to the Drupal 10 site.

Proposed resolution

- Migrate the NID property from Drupal 7 into the Drupal 10 records
- Set the autoincrement NID high enough for records created in the Drupal 10 site to avoid conflicts.

Remaining tasks

MR and test.

📌 Task
Status

Active

Version

1.0

Component

Code

Created by

🇪🇸Spain fjgarlin

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @fjgarlin
  • Merge request !10Preserve nids on node migrations. → (Merged) created by fjgarlin
  • 🇪🇸Spain fjgarlin

    Need to set AUTO_INCREMENT values to avoid conflicts for several tables:
    - node
    - paragraph
    - comments

    Current number of values on some keytables in D7:

    select count(*) from field_collection_item;  // 897697
    select count(*) from node; // 2083370
    select count(*) from node_revision; // 8645435
    select count(*) from field_collection_item_revision; // 1152605
    select count(*) from comment; // 10432900
    
  • 🇪🇸Spain fjgarlin

    Latest IDs (these are moving day by day) as of 2024-10-15 15:40 CEST

    select item_id from field_collection_item order by item_id desc limit 1; // 1677872
    select nid from node order by nid desc limit 1; // 3480812
    select vid from node_revision order by vid desc limit 1; // 13746912
    select revision_id from field_collection_item_revision order by revision_id desc limit 1; // 1973212
    select cid from comment order by cid desc limit 1; // 15815458
    
  • 🇪🇸Spain fjgarlin

    All auto_increment fields from the D10 database:

    SELECT TABLE_NAME, COLUMN_NAME FROM `information_schema`.`COLUMNS` WHERE `EXTRA` = 'auto_increment' AND `TABLE_SCHEMA` = 'db' AND TABLE_NAME NOT LIKE 'migrate%';
    +----------------------------+-------------+
    | TABLE_NAME                 | COLUMN_NAME |
    +----------------------------+-------------+
    | comment                    | cid         |
    | file_managed               | fid         |
    | path_alias_revision        | revision_id |
    | social_auth                | id          |
    | path_alias                 | id          |
    | help_search_items          | sid         |
    | search_api_task            | id          |
    | menu_link_content          | id          |
    | block_content_revision     | revision_id |
    | paragraphs_item            | id          |
    | redirect                   | rid         |
    | queue                      | item_id     |
    | watchdog                   | wid         |
    | crop_revision              | vid         |
    | media                      | mid         |
    | menu_tree                  | mlid        |
    | sequences                  | value       |
    | node                       | nid         |
    | users                      | uid         |
    | taxonomy_term_data         | tid         |
    | paragraphs_item_revision   | revision_id |
    | batch                      | bid         |
    | node_revision              | vid         |
    | taxonomy_term_revision     | revision_id |
    | crop                       | cid         |
    | block_content              | id          |
    | media_revision             | vid         |
    | menu_link_content_revision | revision_id |
    +----------------------------+-------------+
    
    
  • 🇪🇸Spain fjgarlin

    All auto_increment fields from the D7 database:

    SELECT TABLE_NAME, COLUMN_NAME FROM `information_schema`.`COLUMNS` WHERE `EXTRA` = 'auto_increment' AND `TABLE_SCHEMA` = 'migrate' AND TABLE_NAME NO
    T LIKE 'migrate%';
    +----------------------------------------+------------------------------+
    | TABLE_NAME                             | COLUMN_NAME                  |
    +----------------------------------------+------------------------------+
    | metatag_config                         | cid                          |
    | bueditor_editors                       | eid                          |
    | mailchimp_signup                       | mcs_id                       |
    | comment                                | cid                          |
    | file_managed                           | fid                          |
    | access                                 | aid                          |
    | project_composer_namespace_map         | map_id                       |
    | views_data_export                      | eid                          |
    | flagging                               | flagging_id                  |
    | drupalorg_comment_deleted              | cid                          |
    | panels_renderer_pipeline               | rpid                         |
    | og_role                                | rid                          |
    | message_type                           | id                           |
    | field_config                           | id                           |
    | role_activity                          | raid                         |
    | field_config_instance                  | id                           |
    | versioncontrol_repositories            | repo_id                      |
    | captcha_sessions                       | csid                         |
    | versioncontrol_item_revisions          | item_revision_id             |
    | salesforce_mapping_object              | salesforce_mapping_object_id |
    | panels_layout                          | lid                          |
    | profile_field                          | fid                          |
    | imagecache_action                      | actionid                     |
    | blocked_ips                            | iid                          |
    | search_api_task                        | id                           |
    | field_group                            | id                           |
    | taxonomy_term_synonym                  | tsid                         |
    | aggregator_category                    | cid                          |
    | fieldable_panels_panes_revision        | vid                          |
    | project_composer_providers             | provider_id                  |
    | panelizer_defaults                     | pnid                         |
    | locales_source                         | lid                          |
    | project_composer_calculated_dependency | calculated_dependency_id     |
    | user_restrictions                      | urid                         |
    | redirect                               | rid                          |
    | pift_ci_job_result                     | job_result_id                |
    | feeds_log                              | flid                         |
    | ctools_custom_content                  | cid                          |
    | queue                                  | item_id                      |
    | menu_links                             | mlid                         |
    | watchdog                               | wid                          |
    | project_composer_dependency            | dependency_id                |
    | role                                   | rid                          |
    | versioncontrol_event_log               | elid                         |
    | salesforce_mapping                     | salesforce_mapping_id        |
    | project_composer_download_counts       | provider_count_id            |
    | bueditor_buttons                       | bid                          |
    | versioncontrol_sync_log                | slid                         |
    | pift_ci_config                         | config_file_id               |
    | project_composer_provider_includes     | provider_include_id          |
    | search_api_server                      | id                           |
    | taxonomy_vocabulary                    | vid                          |
    | flood                                  | fid                          |
    | field_collection_item                  | item_id                      |
    | sequences                              | value                        |
    | tfa_trusted_browser                    | did                          |
    | node                                   | nid                          |
    | files                                  | fid                          |
    | block                                  | bid                          |
    | bueditor_plus_profiles                 | pid                          |
    | tfa_recovery_code                      | id                           |
    | field_collection_item_revision         | revision_id                  |
    | taxonomy_term_data                     | tid                          |
    | pift_ci_job                            | job_id                       |
    | date_formats                           | dfid                         |
    | image_effects                          | ieid                         |
    | tfa_accepted_code                      | cid                          |
    | search_api_index                       | id                           |
    | page_manager_pages                     | pid                          |
    | package                                | id                           |
    | node_revision                          | vid                          |
    | views_view                             | vid                          |
    | aggregator_feed                        | fid                          |
    | block_custom                           | bid                          |
    | og_membership                          | id                           |
    | taxonomy_term_relation                 | trid                         |
    | imagecache_preset                      | presetid                     |
    | site                                   | sid                          |
    | versioncontrol_auth_account_label      | label_id                     |
    | job_schedule                           | item_id                      |
    | panels_display                         | did                          |
    | message                                | mid                          |
    | pift_ci_notification                   | notification_id              |
    | url_alias                              | pid                          |
    | og_membership_type                     | id                           |
    | page_manager_handlers                  | did                          |
    | image_styles                           | isid                         |
    | flag                                   | fid                          |
    | message_type_category                  | id                           |
    | project_composer_component             | component_id                 |
    | authmap                                | aid                          |
    | contact                                | cid                          |
    | fieldable_panels_panes                 | fpid                         |
    | aggregator_item                        | iid                          |
    | versioncontrol_operations              | vc_op_id                     |
    | salesforce_mapping_object_revision     | revision_id                  |
    | panels_pane                            | pid                          |
    | drupalorg_packaging_job                | job_id                       |
    | pift_ci_label_map                      | label_id                     |
    | versioncontrol_labels                  | label_id                     |
    +----------------------------------------+------------------------------+
    
    
  • 🇪🇸Spain fjgarlin

    Suggested tables and code to change the auto_increment value to:

    $auto_increment_new_values = [
        'node' => 10000000,
        'node_revision' => 20000000,
        'paragraph_item' => 2000000,
        'paragraph_item_revision' => 10000000,
        'comment' => 20000000,
    ];
    foreach ($auto_increment_new_values as $table => $value) {
        \Drupal::database()->query("ALTER TABLE {" . $table . "} AUTO_INCREMENT = " . $value);
    }
    
  • Pipeline finished with Skipped
    6 months ago
    #313379
  • 🇪🇸Spain fjgarlin

    Merged, But totally ok to have code reviews or comments after it, as we can correct anything wrong.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024