Private files corrupted by migration action upgrade_d7_file_private

Created on 12 January 2023, over 1 year ago
Updated 20 June 2023, 12 months ago

Problem/Motivation

The scenario is that we are upgrading/migrating a site from Drupal 7 to Drupal 9.
We operate the web sites ona hosted server. We maintain multiple drupal versions
Running in specific folders, as described in the details below. While running migrate_manifest during the migration action "upgrade_d7_private_files" all
The contents of the private files (used reliably in the D7 context) were overwritten and corrupted. Every private file was overwritten with the same HTML content, regardless of the content prior to this migration step.

Steps to reproduce

Our computing environment is:
Apache version 2.4.54
PHP version 7.4.33
MySQL version 10.5.18-MariaDB
OS is linux

Our live drupal 7 site is running Drupal 7.94
The Drupal 9 site that we are upgrading to is running Drupal 9.5.1

A high level view of our file system, with relevant folders looks like:

/home/myname/drupal_files /* Folder for all Private files for all drupal instances */
/home/myname/public_html/d7site /* Folder for the D7 core, modules, themes, …*/
/home/myname/public_html/d9site /* Folder for the D9 core, modules, themes, …*/

The Drupal 7 database is set up with:
Dbname: myname_db_d7 (built to migrate from drupal 7 to BETA drupal 9)
user: myname_shiva
passwd: 77777!
 
The drupal 9 database is set up with:
Dbname: myname_beta9
User: myname_niner
Passwd: 99999!

Consider the settings in /home/public_html/d9site/sites/default/settings.php
The private files setting in the D9 settings.php file is:
$settings['file_private_path'] = '/home/myname/drupal_files';

The "drupal_files" folder contains many documents. Let's consider one pdf file at:
/home/myname/drupal_files/abc.pdf

The database declaration in the D9 settings.php file contains this declaration:

$databases['migrate']['default'] = array (
'database' => 'myname_db_d7',
'username' => 'myname_shiva',
'password' => '77777!,
'prefix' => '',
'host' => 'localhost',
'port' => '3306',
'namespace' => 'Drupal\\mysql\\Driver\\Database\\mysql',
'driver' => 'mysql',
);

Modules that have been added to the Drupal 9 site include a set of migration modules:
composer require 'drupal/backup_migrate:^5.0'
composer require 'drupal/migrate_upgrade:^4.0'
composer require 'drupal/migrate_tools:^6.0'
composer require 'drupal/migrate_manifest:^3.0'

each of these modules is enabled.

After generating a list of migration actions with the drush command operating in the D9site context:
drush migrate-upgrade –legacy-db-key=migrate –legacy-root=http://myname.org/d7site --configure-only

One of the first migration actions generated is:
upgrade_d7_file_private

When I used drush migrate-manifest with an active line for
upgrade_d7_file_private
During the execution of this migration, All files in the private file folder /home/myname/drupal_files/*
All become corrupted, overwritten with some html formatted text.

I believe the same thing happens if I use the command:
Drush migrate-import upgrade_d7_file_private

Proposed resolution

The migration system should NOT over write files in the "private files" folder.

Remaining tasks

feel free to contact me if you have any questions about how to reproduce this error.

User interface changes

API changes

Data model changes

Release notes snippet

💬 Support request
Status

Closed: works as designed

Version

9.5

Component
Migration  →

Last updated about 12 hours ago

Created by

🇺🇸United States kenbron

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Status changed to Closed: works as designed 12 months ago
  • 🇺🇸United States mikelutz Michigan, USA

    Closing this, as there is no defect with the migrate api here. The plugins and tools used by core are designed and and documented to work for migrations into a brand new freshly installed drupal installation with an empty file directory, as those plugins and migration configurations are designed to be very aggressive in forcing the input data to match the old data. The error messages that overwrote the files may be due to them being inaccessible at the migrations source_base_path, or trying to open the file streams for reading or writing at the same time because the source and destination paths were the same, I'm not sure. Regardless setting the source and destination to the same directory is not an expected or supported use of that plugin. Using any of the core d7 migration plugins to migrate into a site with any data in it, be it files or content is likely to cause that data to be overwritten. This is desirable and expected behavior. The migrate api is capable of copying database records without overwriting files, but if you configure it to use the file copy plugin and point that plugin to a directory with existing files, then they can and are expected to be overwritten.

Production build 0.69.0 2024