Regression AND bug: SQLite source and id map tables aren't joinable anymore, but SqlBase::mapJoinable() is unaware of this

Created on 24 January 2022, over 2 years ago
Updated 26 June 2023, about 1 year ago

Problem/Motivation

Drupal core's Migrate API has a regression since #3124674: Refactor functionality for the SQLite driver to stop migration testing usage of the 'extra_prefix' connection option β†’ was committed to core, because database tables of a SQLite source and SQLite ID map tables aren't joinable anymore.

Unfortunately, Migrate Drupal UI tests are performed with SQLite database driver aren't reveal this issue, because these tests are setting up a "not joinable" source and destination databases.
On the other hand, MigrateTestBase's createMigrationConnection manually attaches the destination SB to the source, so it hides this regression/bug. There isn't any (functional) migrate thing in core that calls Connection::attachDatabase(), so I'm pretty sure it shouldn't be called in tests.

I will upload a functional regression test which uses Migrate API, initializes the source plugin instance of d7_user_role (and then checks whether the map table was created), then verifies that the source plugin of d7_user_role tells us that the source and the id map tables are joinable, and then finally tries to get the source plugin instance's source values.

This will be a new test, so it can be applied on 9.2.x and so (hopefully) you will see that it works well with that core version.

Discovered in #3218083-14: Handle long media source field names β†’ .

Steps to reproduce

  1. Install Drupal 7 with a SQLite database with standard profile:

    In settings.php you should have something like this:

    // Default (destination) connection:
    $databases['default']['default'] = [
      'driver' => 'sqlite',
      'database' => '<absolute_path_to>/sqlite.db',
    ];
       
  2. Define an additional connection in the settings.php with key 'fixture_connection': copy-past default connection, and add a prefix:
    // Fixture connection:
    $databases['fixture_connection']['default'] = [
      'driver' => 'sqlite',
      'database' => '<absolute_path_to>/sqlite.db',
      'prefix' => 'test123',
    ];
       
  3. Import Drupal core's Drupal 7 migrate database fixture into 'fixture_connection'. (So: follow steps at https://www.drupal.org/docs/drupal-apis/migrate-api/generating-database-... β†’ )
  4. Optional: remove the fixture connection from settings.php.
  5. Install Migrate Drupal UI, visit /upgrade, and at the credentials form select SQLite. Fill in the path to the SQLite db file and the prefix of the fixture connection.
  6. Execute the upgrade.

Expected result: Upgrade executed without errors on every 9.* core versions.
Actual result on 9.3.x or on 9.4.x: Upgrade failed, only those migrations are executed which have no-sql source plugin.

Proposed resolution

  1. Restore the original functionality.
  2. A: Instead of removing or breaking anything, just warn users that SQLite source and destination tables won't be joinable in the future major release of core.
  3. B: Ensure that the migrate subsystem attaches the id map's database to the source migration db.

Remaining tasks

Depends.

User interface changes

Nothing.

API changes

Restored API, SQLite source and id map tables are joinable again.

Data model changes

Nothing.

Release notes snippet

Nothing.

πŸ› Bug report
Status

Closed: cannot reproduce

Version

10.0 ✨

Component
MigrationΒ  β†’

Last updated about 1 hour ago

Created by

πŸ‡­πŸ‡ΊHungary huzooka Hungary πŸ‡­πŸ‡ΊπŸ‡ͺπŸ‡Ί

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡ΈUnited States mikelutz Michigan, USA

    I'm trying to resolve or close old migrate issues. It's been about a year since we've heard anything on this. I'm going to close this as 'cannot reproduce' but feel free to reopen it if the OP or original reviewers can comment as to whether the latest patch sufficiently solves the issue at hand.

Production build 0.69.0 2024