Allow /dev/null as the source path

Created on 13 October 2024, 4 months ago

Problem/Motivation

If I use this module with the migrate_source_ui module, then I can upload a CSV file and that module will override the path in the source plugin configuration. So far, so good.

Next, I run a second migration using this module and migrate_suorce_ui. I want to use migration_lookup to import related items from the first migration. This does not work, since migrate_source_ui does not override the source plugin configuration from the first migration. Unless my migration specifies a valid path to a CSV file, the source plugin throws an exception when it is instantiated for the lookup.

Steps to reproduce

Proposed resolution

Allow the following in the migration configuration:

source:
  plugin: csv
  path: /dev/null
  ids:
    - id

Unless path is overridden, the empty file means that the source plugin will provide 0 rows.

Remaining tasks

User interface changes

API changes

Data model changes

✨ Feature request
Status

Active

Version

3.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @benjifisher
  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I am adding ✨ Allow a callback function for the Source path parameter Active as a related issue. I do not see an easy way to solve this problem with the proposed resolution from that issue, but maybe there is a way.

  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    MR 14 is a simple, 1-line change.

  • Pipeline finished with Failed
    4 months ago
    Total: 262s
    #308720
  • Pipeline finished with Failed
    4 months ago
    Total: 139s
    #309384
  • Pipeline finished with Failed
    4 months ago
    Total: 137s
    #309435
  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I must have been doing something wrong, because the one-line change seemed to be working yesterday. On further testing, it does not.

    The current MR adds an empty file (empty.csv) to the codebase. If the path is set to the specific string '/dev/null', then the constructor replaces that (in $this->configuration) with the path to the empty file and also sets header_offset. The result is that the source plugin returns zero rows, without errors, warnings, nor exceptions.

    Other benefits.

    With the feature branch, I can roll back a migration:

    $ ddev drush mr import_csv_terms
     [notice] Rolled back 7 items - done with 'import_csv_terms'
    $ ddev drush ms import_csv_terms
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- --------------- 
      Group                     Migration ID       Status   Total   Imported   Unprocessed   Message Count   Last Imported  
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- --------------- 
      Import CSV (import_csv)   import_csv_terms   Idle     0       0          0             0                              
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- --------------- 
    

    With the 8.x-3.x branch, I cannot roll back a migration unless it specifies an actual path:

    $ ddev drush mr import_csv_terms
    
    In UnavailableFeature.php line 40:
                                        
      stream does not support seeking.  
                                        
    
    Failed to run drush mr import_csv_terms: exit status 1
    

    With the feature branch, when I check the status of my migrations, I get this:

    $ ddev drush ms
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- --------------------- 
      Group                     Migration ID       Status   Total   Imported   Unprocessed   Message Count   Last Imported        
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- --------------------- 
      Import CSV (import_csv)   import_csv_terms   Idle     0       7          -7            0               2024-10-14 13:40:21  
      Import CSV (import_csv)   import_csv_users   Idle     0       3          -3            0               2024-10-14 13:40:21  
      Import CSV (import_csv)   import_csv_nodes   Idle     0       3          -3            0               2024-10-14 13:40:21  
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- ---------------------
    

    With the 8.x-3.x branch, I get errors (but the same information):

    $ ddev drush ms
     [error]  Could not retrieve source count from import_csv_terms: stream does not support seeking. 
     [error]  Could not retrieve source count from import_csv_users: stream does not support seeking. 
     [error]  Could not retrieve source count from import_csv_nodes: stream does not support seeking. 
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- --------------------- 
      Group                     Migration ID       Status   Total   Imported   Unprocessed   Message Count   Last Imported        
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- --------------------- 
      Import CSV (import_csv)   import_csv_terms   Idle     N/A     7          N/A           0               2024-10-14 13:40:21  
      Import CSV (import_csv)   import_csv_users   Idle     N/A     3          N/A           0               2024-10-14 13:40:21  
      Import CSV (import_csv)   import_csv_nodes   Idle     N/A     3          N/A           0               2024-10-14 13:40:21  
     ------------------------- ------------------ -------- ------- ---------- ------------- --------------- ---------------------
    
  • Pipeline finished with Failed
    4 months ago
    Total: 137s
    #309580
  • Pipeline finished with Skipped
    about 2 months ago
    #379770
  • First commit to issue fork.
  • Issue was unassigned.
  • Status changed to Needs review about 2 months ago
  • heddn Nicaragua

    Thanks for the contributions. This makes a lot of sense.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024