Fully support remote stream wrappers in FileCopy plugin

Created on 11 June 2021, over 3 years ago
Updated 25 April 2024, 7 months ago

Problem/Motivation

Currently in the fileCopy plugin realpath() is called inside getDirectory() when a path is down to the root ('public://', 'private://', etc) .

On a remote StreamWrapper such as S3FS realpath() will return false which will cause issues later on.

According to fileCopy's source it doesn't appear it was intended to be used with remote stream wrappers so filing this as a feature.

The scenario that brought this up was a user migrating from a local public:// filesystem to one where using the s3fs module public:// was provided by an S3 bucket.

Steps to reproduce

Migrate when using a non local StreamWrapper (such as s3fs), files in a directory will migrate but files in the root of the StreamWrapper will not.

Proposed resolution

I believe this could be added by checking if realpath() returns false and if so return $dir unmodified inside of getDirectory()

Remaining tasks

User interface changes

None expected

API changes

None expected

Data model changes

None expected

Release notes snippet

✨ Feature request
Status

Active

Version

11.0 πŸ”₯

Component
MigrationΒ  β†’

Last updated 1 day ago

Created by

πŸ‡ΊπŸ‡ΈUnited States cmlara

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡«πŸ‡·France pacproduct

    We faced this issue while migrating a Drupal 6 site to Drupal 10.

    The Drupal 6 site uses the standard local file system, but the new one has been configured to store its private files to a remote S3 server with s3fs β†’ .

    I'm not sure why "realpath" is used initially when the resulting directory is only a scheme (e.g. 'private://'), neither why it's a problem to "trip up certain file API functions" as documented in \Drupal\migrate\Plugin\migrate\process\FileCopy::getDirectory().

    We implemented @cmlara's suggestion in a custom Plugin that extends core class FileCopy. I don't know if it's the best approach but it works for us, so maybe it will for some other people too. Here is the source code:

    namespace Drupal\your_module\Plugin\migrate\process;
    
    use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
    use Drupal\migrate\Plugin\migrate\process\FileCopy;
    
    /**
     * Copies a file, in a way compatible with remote file systems.
     * 
     * This class extends the core "file_copy" plugin, but handles cases where
     * "realpath" can't work.
     * See file_copy's documentation for how to use it.
     *
     * Example:
     *
     * @code
     * process:
     *   path_to_file:
     *     plugin: remote_compatible_file_copy
     *     source:
     *       - /path/to/file.png
     *       - public://new/path/to/file.png
     * @endcode
     *
     * @see \Drupal\migrate\Plugin\MigrateProcessInterface
     *
     * @MigrateProcessPlugin(
     *   id = "remote_compatible_file_copy"
     * )
     */
    class RemoteCompatibleFileCopy extends FileCopy implements ContainerFactoryPluginInterface {
      /**
       * Returns the directory component of a URI or path.
       *
       * Unlike `FileCopy::getDirectory`, if the path to the directory ends up
       * being the scheme only (e.g.: 'private://') and if `realpath` fails at
       * determining the path, we do not return FALSE but the scheme alone.
       *
       * @param string $uri
       *   The URI or path.
       *
       * @return string
       *   The directory component of the path or URI.
       */
      protected function getDirectory($uri): string {
        $dir = $this->fileSystem->dirname($uri);
        if (str_ends_with($dir, '://')) {
          $realpath = $this->fileSystem->realpath($dir);
          if (FALSE !== $realpath) {
            return $realpath;
          }
        }
        return $dir;
      }
    
      /**
       * Determines if the source and destination URIs represent identical paths.
       *
       * Unlike `FileCopy::isLocationUnchanged` we always return FALSE if one of
       * the given paths cannot be determined.
       *
       * @param string $source
       *   The source URI.
       * @param string $destination
       *   The destination URI.
       *
       * @return bool
       *   TRUE if the source and destination URIs refer to the same physical path,
       *   otherwise FALSE.
       */
      protected function isLocationUnchanged($source, $destination): bool {
        $sourceRealPath = $this->fileSystem->realpath($source);
        $destinationRealPath = $this->fileSystem->realpath($destination);
    
        if (FALSE === $sourceRealPath || FALSE === $destinationRealPath) {
          return FALSE;
        }
    
        return $sourceRealPath === $destinationRealPath;
      }
    }
    
  • πŸ‡ΊπŸ‡ΈUnited States mikelutz Michigan, USA
Production build 0.71.5 2024