Allow wildcard for URL source plugin

Created on 6 November 2017, over 6 years ago
Updated 1 May 2024, about 2 months ago

Problem/Motivation

Not always is possible to know the name of all the files to import. For example, a massive continuous exporter adding a file for a content node, with its identifier as file name.

Proposed resolution

Implement support for wildcards in the urls configuration key. With this we can target the files based on a definition criteria depending on the file name/extension.

This patch adds support for a new key to be set as true if want to check for wildcard (with glob php function) for the elements defined in urls

Remaining tasks

  • Community test & QA
  • Document the new wildcard option for the module
✨ Feature request
Status

Needs work

Version

6.0

Component

Plugins

Created by

πŸ‡ͺπŸ‡ΈSpain quiron Barcelona

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • First commit to issue fork.
  • πŸ‡¬πŸ‡§United Kingdom joachim

    Made a MR with a few small fixes.

  • Pipeline finished with Canceled
    about 2 months ago
    Total: 37s
    #161634
  • Pipeline finished with Failed
    about 2 months ago
    Total: 227s
    #161636
  • Pipeline finished with Failed
    about 2 months ago
    Total: 239s
    #161650
  • πŸ‡¬πŸ‡§United Kingdom joachim

    re: #28, allowing callbacks might be useful too, but globs are a useful tool that we have in a filesystem which we don't have for URLs, which make a declarative definition possible. So I think it makes sense to have them too.

  • πŸ‡¬πŸ‡§United Kingdom joachim

    Found a problem when the URL doesn't use a stream wrapper like public:// and is intead an absolute filepath.

    Working on fixing it...

  • Pipeline finished with Failed
    about 2 months ago
    #161679
  • Status changed to Needs work about 2 months ago
  • πŸ‡¬πŸ‡§United Kingdom joachim

    re #28 again... ah now I've read more of this patch and the patch at the other issue, I see how they could work together.

    ✨ Allow callback for Url source, and single item Json plugin RTBC should be changed so it defers executing the callback, and that's when this MR should run the globbing too -- in neither case should we be spookily changing the plugin's configuration, that's just confusing DX!

  • Pipeline finished with Failed
    about 2 months ago
    Total: 302s
    #161696
Production build 0.69.0 2024