Make fields() method work with nested fields

Created on 18 August 2023, over 1 year ago
Updated 27 May 2024, 6 months ago

Problem/Motivation

The method \Drupal\migrate_source_graphql\Plugin\migrate\source\GraphQL::fields() method does not work properly. At least not with a more complex query like this example

  query:
    search:
      listings:
        arguments:
          locale: nl_BE
          metadata:
            size: 400
          publication:
            changedDateFrom: '2023-08-12T00:00:00.000Z'
        fields:
          - listings:
            - id
            - details:
              - vehicle:
                - classification:
                  - make:
                    - formatted
                  - model:
                    - formatted
                  - modelVersionInput
                - numberOfDoors

I can access the fields in the process phase of the migration and map them to destination properties. But drush migrate:fields-source does not return the complete list of available fields.

I have marked this issue as minor since it seems that drush migrate:fields-source is the only command that uses the method.

Steps to reproduce

TBD

Proposed resolution

Add recursion to the discovery of fields so that drush migrate:fields-source can return a complete overview of the fields available in the source.

Remaining tasks

-
-
-

User interface changes

drush migrate:fields-source will return a complete list of available fields.

API changes

N/A

Data model changes

N/A

🐛 Bug report
Status

Fixed

Version

2.0

Component

Code

Created by

🇳🇱Netherlands ricovandevin

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @ricovandevin
  • @ricovandevin opened merge request.
  • Status changed to Needs review over 1 year ago
  • 🇳🇱Netherlands ricovandevin

    With the example

      query:
        search:
          listings:
            arguments:
              locale: nl_BE
              metadata:
                size: 400
              publication:
                changedDateFrom: '2023-08-12T00:00:00.000Z'
            fields:
              - listings:
                - id
                - details:
                  - vehicle:
                    - classification:
                      - make:
                        - formatted
                      - model:
                        - formatted
                      - modelVersionInput
                    - numberOfDoors
    

    the command drush migrate:fields-source will return

     -------------------------------------------------- ------------------- 
      Field name                                         Description        
     -------------------------------------------------- ------------------- 
      id                                                 id                 
      details                                            details            
      details/vehicle                                    vehicle            
      details/vehicle/classification                     classification     
      details/vehicle/classification/make                make               
      details/vehicle/classification/make/formatted      formatted          
      details/vehicle/classification/model               model              
      details/vehicle/classification/model/formatted     formatted          
      details/vehicle/classification/modelVersionInput   modelVersionInput  
      details/vehicle/numberOfDoors                      numberOfDoors      
     -------------------------------------------------- -------------------
    

    after this MR.

  • 🇮🇹Italy reinchek Napoli 🌋, 🇮🇹

    Hi @ricovandevin, thanks!
    I reviewed and locally tested your MR. It works well!
    I would like to take advantage of this upcoming release to also add some module's tests. At least the kernel-type one; If I have the time, however, I implement unit ones as well.

    I try to do this tomorrow.
    If I find that I cannot do this I will proceed with the merge of your MR.

    Thanks ;)

  • 🇮🇹Italy reinchek Napoli 🌋, 🇮🇹

    Hi @ricovandevin, i've merged your MR inside a new @dev branch (you can found it there).
    Just to avoid to create too much minimum module's versions.
    So, in this way we can collect more features in a single release :)

    Thanks again. I'm extending your data_key PROPERTY_SEPARATOR logic introducing "%" (ex: `data_key: data/%/user` ) symbol that means: maps all data array's values getting for each the user object. For easy understanding, the code:

          const DATA_KEY_INDEXES = '%';
          // ...
          // ...
          $getIndexes = false;
          foreach ($propertyPath as $path) {
            if ($path === self::DATA_KEY_INDEXES) {
              $getIndexes = true;
              continue; // goto the next path item, with $getIndexes eq true.
            }
            if ($getIndexes) {
              $results = array_map(function ($item) use ($path) {
                return $item->{$path};
              }, array_values($results));
              $getIndexes = false;
            } else {
              $results = $results->{$path} ?? [];
            }
          }
    

    so if we has a response from graphql server like this:

    $data => array (
        0 =>
            (object) array(
                'user' =>
                    (object) array(
                        'id' => '6',
                        'username' => 'Leopoldo_Corkery',
                        'name' => 'Mrs. Dennis Schulist',
                        'email' => 'Karley_Dach@jasper.info',
                    ),
            ),
    );
    

    The results will be remapped as:

    $results => array (
      0 => 
      (object) array(
         'id' => '6',
         'username' => 'Leopoldo_Corkery',
         'name' => 'Mrs. Dennis Schulist',
         'email' => 'Karley_Dach@jasper.info',
      ),
    )
    

    This could provide more flexibility to the module.
    What do you think about?
    Soon i will push the code in a new MR.

  • Status changed to Fixed over 1 year ago
  • 🇮🇹Italy reinchek Napoli 🌋, 🇮🇹
  • Status changed to RTBC over 1 year ago
  • 🇮🇹Italy reinchek Napoli 🌋, 🇮🇹
  • 🇮🇹Italy reinchek Napoli 🌋, 🇮🇹
  • 🇳🇱Netherlands ricovandevin

    @reinchek Your suggestion for introducing "%" looks great. I will test it as soon as I have some spare time!

  • 🇮🇹Italy reinchek Napoli 🌋, 🇮🇹

    @ricovandevin, thanks!
    You can find all in this MR:
    - https://git.drupalcode.org/project/migrate_source_graphql/-/merge_requests/10

    The MR was created on top of current (2.x) dev branch. Inside you can find:
    - PROPERTY_SEPARATOR logic
    - possibility to use the '%' placeholder to get all array's items
    - migrate_source_graphql_test sub-module which setup some Kernel tests (runs some basic migration cases): needs work.

    Thanks again

  • 🇮🇹Italy reinchek Napoli 🌋, 🇮🇹

    Hi,

    i've merged https://git.drupalcode.org/project/migrate_source_graphql/-/merge_reques... inside 2.x (dev) branch.
    Now, dev also contains the following features:
    - PROPERTY_SEPARATOR logic
    - possibility to use the '%' placeholder to get all array's items
    - migrate_source_graphql_test sub-module which setup some Kernel tests (runs some basic migration cases): needs work.

    Thanks.

  • Status changed to Fixed 6 months ago
  • 🇮🇹Italy reinchek Napoli 🌋, 🇮🇹

    Fixed and merged in 2.x (dev) branch.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024