Additional source plugin filtering

Created on 9 February 2024, about 1 year ago
Updated 21 February 2024, about 1 year ago

Problem/Motivation

Some idea in this realm was initiated in Add status filter to migration source Needs review , but I'd like to see even more features exposed from the GC List Items API exposed in the source plugin.

Steps to reproduce

Proposed resolution

Config options are always available in a source plugin and are statically set during construction. Add additional config options for more filtering options. Then use those filtering options when passing along to getItems(). Don't forget to add config schema for this. See 🐛 Config schema fixes RTBC .

Remaining tasks

User interface changes

API changes

Data model changes

📌 Task
Status

RTBC

Version

6.0

Component

Code

Created by

heddn Nicaragua

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @heddn
  • First commit to issue fork.
  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.5 + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    5 pass
  • Pipeline finished with Success
    about 1 year ago
    Total: 268s
    #96186
  • Status changed to Needs review about 1 year ago
  • 🇵🇪Peru marvil07

    I have added a possible implementation on a new merge request.
    From the description there:

    The underlying used library receives and passes the request query options to the List items request on the API.

    In the same spirit, after exposing a way to pass directly those options in the migrate source plugin configuration, filtering is expanded to support any of the request query options in the API.

    New commits on 3420528-add-source-filters topic branch and related new MR !14

    • 102e629 Allow GC client list items request query options override
    • 362f056 Declare new schema for migrate_plus.source.gathercontent_migration configuration
    • d76227d Directly use migrate source plugin configuration
    • b6de622 Include list items request query parameters on configuration schema
  • Status changed to Needs work about 1 year ago
  • heddn Nicaragua

    Posted minor feedback on MR.

  • Open in Jenkins → Open on Drupal.org →
    Core: 9.5.5 + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    5 pass
  • Pipeline finished with Success
    about 1 year ago
    Total: 185s
    #99575
  • Status changed to Needs review about 1 year ago
  • 🇵🇪Peru marvil07

    @heddn, I have added types and labels to the new client_options key.

    Please notice that even if API documentation uses types like boolean, or array of type; at the end they are represented as a comma separated values string.
    Hence, I am reflecting that instead, given the added option tries to reflect directly what will be passed to the underlying library, and ultimately to the API.

    New commits added to 3420528-add-source-filters topic branch and related MR !14

    - 53dcd97 Add types and labels to client_options mapping on source schema

  • Status changed to RTBC about 1 year ago
  • heddn Nicaragua

    It works. Really well. LGTM.

Production build 0.71.5 2024