Rearrange Url plugin configuration

Created on 7 July 2016, almost 8 years ago

Updated 1 May 2024, about 2 months ago

Right now, the structure of the Url source plugin configuration is:

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls: /migrate_example_advanced_position?_format=json
  authentication: # See 
            
              #2761489: Provide authentication plugins to HTTP fetcher →
            
  item_selector: 1
  fields:
    ...
  ids:
    ...

Everything, even options specific to either the fetcher or parser, is at the source plugin configuration level. Ideally, I'd like things to be broken down cleanly between them, like

source:
  plugin: url
  fetcher:
    plugin: http
    authentication: # See 
            
              #2761489: Provide authentication plugins to HTTP fetcher →
            
  parser:
    plugin: json
    # One might expect this to be a fetcher option, but the parser controls the fetcher, determining
    # when to get the the next URL in the list, and the fetcher is designed for one URL at a time.
    urls: /migrate_example_advanced_position?_format=json
    item_selector: 1
  fields:
     ...
  ids:
    ...

Two issues with that:

We already have two cases of parser plugins that subsume the fetching functionality (xml and soap), so the fetcher isn't actually used. That being said, now that I think about it if the parser plugin has access to the whole source configuration it can peek at the fetcher config, even if the actual fetcher plugin isn't being used.
This is obviously disruptive for anyone already working with this plugin. Although, I'm wondering if the source plugin could recognize the "old" structure (seeing 'urls' at the top level) and rearrange it into the "new" structure, which would be used internally?

Thoughts?

📌 Task

Status

Active

Version

6.0

Component

Plugins

Created by

🇺🇸United States mikeryan Murphysboro, IL, USA

Live updates comments and jobs are added and updated live.

Incomplete comments

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Comment about 2 months ago →
🇬🇧United Kingdom joachim
The source configuration is going to get more complicated with changes like ✨ Allow wildcard for URL source plugin Needs work and ✨ Allow callback for Url source, and single item Json plugin RTBC .

contrib.social Blog FAQ Discussions

Production build 0.69.0 2024