[meta] Feeds 8.x roadmap

Created on 4 April 2013, about 11 years ago
Updated 5 September 2023, 10 months ago

The current Feeds release works great. There are a few bugs and features missing that are blocking a stable release.

We aim to create new Feeds releases every 3-4 months.

Feeds issues series

On this page:

8.x-3.x

This list groups issues that are either requested a lot in the issue queue or are features that are not yet ported from the D7 version of Feeds.

DONE

8.x-3.0-beta1 was released on February 24, 2022. It contains the previously missing UI for managing custom sources - which was needed for Feeds Extensible Parsers to add options for their custom source types.

These issues were beta targets, but did not make it into the first beta:
🐛 A translation already exists for the specified language Active
#2998037: Allow more field types to be used as reference filter fields
#3073275: Mapping to books

DONE

Before addressing the stable release blockers, we are first focusing on making Feeds compatible with Drupal 10. So hopefully that work is finished before the first release of Drupal 10.

This does mean Drupal 8 support will get dropped. Because I think that having a 8.x- prefix on a release that does not support Drupal 8 looks weird, we're opening a 4.x branch soon. The deprecated class "FeedsAnnotationFactory" will also be dropped in that branch:
📌 Remove deprecated FeedsAnnotationFactory class Needs review
Existing Feeds plugins that haven't been updated since FeedsAnnotationFactory was deprecated (which happened more than 2 years ago in 8.x-3.0-alpha6 ), may stop working. This is for example the case for the FeedsTarget plugin in the Paragraphs module.

Paragraphs FeedsTarget plugin would need to:

  • Remove the "arguments" section from the annotation;
  • Implement Drupal\Core\Plugin\ContainerFactoryPluginInterface and use the createInstance() method to inject dependencies.

3. Stable release blockers

The following bugs are important to fix before releasing a stable version as they could introduce API changes or additions:
📌 Missing temporary files in load balanced environments Fixed
#3155929: Http Fetcher receiving 304s Do not clear out their queue items.
🐛 FetcherResult::checkFile() throws \RuntimeException, which is never caught Needs review
📌 Remove deprecated FeedsAnnotationFactory class Needs review

4. File target issues

There are several issues reported for the File target.
Right now - if you want to import files - they need to come from a HTTP source.

We want to fix the following things for the file target:

  • Support stream wrappers (like public:// and private://)
  • Support relative server paths
  • Support downloading files where the url doesn't specify an extension
  • Properly cleaning up previously imported files (when replaced with an other file)
  • New option to import by file url that has the current functionality and hide the reference by field if checked
  • When reference by file ID, expect a file ID as input
  • When reference by UUID, expect a UUID as input
  • Repair “existing files” setting.

See issues tagged with Feeds file target

5. Media support

Feeds doesn't support the core Media module completely.
Add a mapping target to media field Needs work

6. Append values workflow

When updating existing entities, Feeds always empty the fields first before writing new values to them. This way empty values can be imported properly.
In some cases you don't want Feeds to empty the values, but instead appending them.
Through out the history of Feeds, this feature has been requested many many times. For D7 we decided to not implement this as it would become too complicated to support. For D8 it might be doable to implement this, now that almost everything are classed objects.
#3136170: handy way of preparing the CSV source?

Historical D7 issues:
#2071095: Need a way to add values to multi-value fields if target entity exists (update existing)
#2059995: Where to access the original value of the entity field to be imported
#2273299: How to retain existing values and append new values in hook_feeds_set_target
#1871970: Getting all the values to the same field from different lines of CSV
#2490840: Multiple records in one import overwrite each other
#2581611: Adding field items (instead of replacing) on update of existing nodes
#1670230: Appending multiple values in one taxonomy reference field
#1789670: Update Node values without needing to include all values in Feed
#3012255: Mapping multiple variations to one product only creates one variation per product.

7. Feeds Import Preview

For the D7 version of Feeds we can see the parsed result with the Feeds Import Preview module, which is very useful for debugging imports. For D8 there's something like this in the sandbox module Feeds DEV, though it's less user friendly and when using it in combination with the HTTP Fetcher, it sometimes doesn't update due to the HTTP Fetcher aborting the fetch when the source returns a 304 status code.
#2141945: Simulate & preview a feed import

8. Multilingual support

In Feeds 8.x-3.0-alpha7 multilingual support was mostly done, but there are a few issues not fixed yet:
🐛 A translation already exists for the specified language Active
Add mapping target for config entity 'language' Active
#3139441: could not update term that is created by feeds

See also Feeds issues tagged with multilanguage .

9. Other

In 2017 we did a survey for Feeds. Based on that, the following features were also marked as important:

  • Support for huge data files
  • Chained imports
  • Rollback imports
  • Keep same content ids as in the source
  • Analytic report of import process

Other interesting issues:
#3173024: Add a generic field target

Feeds Tamper

There is a fatal error when using a certain combination of Tamper plugins, blocking a stable release:
🐛 Catch TamperException to prevent import from crashing Needs review
#3063055: Allow parsers and event subscribers to mark an item as invalid

Tamper

There are a few Tamper plugins left to port from Feeds Tamper 7.x-1.x:
#2958013: [META] port remaining feeds tamper plugins

Feeds Extensible Parsers

A few features are left to be ported from the D7 version:

Feeds 7.x-2.x

It's been a while since the last D7 release for Feeds. It would be nice to get it compatible with recent PHP versions. The dev version also has some unfinished bug fixes: a somewhat larger bug was divided into chunks, but not all chunks were done yet.

  • PHP 7.2+ compatibility
  • Large file import
  • Validation issues

For more information, see #2911291: Plan for Feeds 7.x-2.0-beta5 release

Feeds Migrate

Most important issue to fix first:
#3002362: Foundation: Feeds migrate Processors/Tamper UI

Next in line:
Migrate Preview

Core issues to be fixed:
📌 Create a trait and base class to implement \Drupal\Component\Plugin\ConfigurableInterface Needs work
#2937177: [PP-1] Migrate plugin base classes should implement ConfigurablePluginInterface

See also:
🌱 [META] Feeds_migrate Road Map Active
#2903175: [Policy, No patch] Consider adopting parts of migrate system in the underlying parts of feeds

Commerce Feeds

#3074303: Automatically set default store for products if source doesn't provide a value

Feeds vs Feeds Migrate

Feeds 8.x-3.x actively being developed until at least the release of Drupal 10
The Feeds Migrate project is experimenting with bringing Feeds UI features to Migrate.

It will take a while before Feeds Migrate's feature set will reach the same level as Feeds 8.x-3.x.

Development of bug fixes and features will continue on the 8.x-3.x version of feeds for the next few years until Feeds Migrate is mature and feature development of feeds can stop.

Feeds will at least be ported to Drupal 10. If it will be ported to Drupal 11 is uncertain at this moment (March 30, 2021).

Feeds Migrate may eventually become the Feeds 4.x version. However, if Drupal 10 introduces any disruptive changes requiring Feeds to make backwards compatibility breaks, there could be a Feeds 4.x version that is 99% the same as the 8.x-3.x version, but with support for Drupal 10. In that case Feeds Migrate may become Feeds 5.x instead.

🌱 Plan
Status

Active

Version

3.0

Component

Code

Created by

🇺🇸United States Dane Powell

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇳🇱Netherlands MegaChriz

    Feeds multilingual issues are now tagged with multilanguage .

  • Any word on php 8.2 compatibility?

  • 🇳🇱Netherlands MegaChriz

    @tonytheferg
    The latest Feeds release is expected to be compatible with Drupal 10 + PHP 8.2. Tests are passing on that combination. Previously, there were issues installing Feeds on PHP 8.2 because a dependency was not declared compatible with PHP 8.2 yet. That issue has now been resolved in the dependency itself. No changes in Feeds were required afterwards, so that's why the current latest release should be PHP 8.2 compatible now. See 📌 Add PHP 8.2 support Fixed for more information.

  • Huh. I tried yesterday and got blocked by composer dependency.

  • 🇹🇷Turkey Orkut Murat Yılmaz

    @tonytheferg, this is my composer output:

    $ composer require 'drupal/feeds:^3.0@beta'                                                                                                           ✔ 
    Info from https://repo.packagist.org: #StandWithUkraine
    ./composer.json has been updated
    Running composer update drupal/feeds
    Loading composer repositories with package information
    Updating dependencies
    Lock file operations: 4 installs, 0 updates, 0 removals
      - Locking drupal/feeds (3.0.0-beta4)
      - Locking laminas/laminas-escaper (2.12.0)
      - Locking laminas/laminas-feed (2.21.0)
      - Locking laminas/laminas-stdlib (3.17.0)
    Writing lock file
    Installing dependencies from lock file (including require-dev)
    Package operations: 4 installs, 0 updates, 0 removals
      - Installing laminas/laminas-stdlib (3.17.0): Extracting archive
      - Installing laminas/laminas-escaper (2.12.0): Extracting archive
      - Installing laminas/laminas-feed (2.21.0): Extracting archive
      - Installing drupal/feeds (3.0.0-beta4): Extracting archive
    5 package suggestions were added by new dependencies, use `composer suggest` to see details.
    Generating autoload files
    60 packages you are using are looking for funding.
    Use the `composer fund` command to find out more!
    No security vulnerability advisories found
    

    Also, output of my drush pml is:

    $ drush pml
     --------------------------------- ------------------------------------------------------------------------------------ ---------- ---------------- 
      Package                           Name                                                                                 Status     Version         
     --------------------------------- ------------------------------------------------------------------------------------ ---------- ---------------- 
      Core                              Actions UI (action)                                                                  Disabled   10.1.2          
      Core (Experimental)               Announcements (announcements_feed)                                                   Disabled   10.1.2          
      Core                              Automated Cron (automated_cron)                                                      Enabled    10.1.2          
      Core                              Ban (ban)                                                                            Enabled    10.1.2          
      Web services                      HTTP Basic Authentication (basic_auth)                                               Disabled   10.1.2          
      Core                              BigPipe (big_pipe)                                                                   Enabled    10.1.2          
      Core                              Block (block)                                                                        Enabled    10.1.2          
      Core                              Block Content (block_content)                                                        Enabled    10.1.2          
      Core                              Book (book)                                                                          Disabled   10.1.2          
      Core                              Breakpoint (breakpoint)                                                              Enabled    10.1.2          
      Core                              CKEditor 5 (ckeditor5)                                                               Enabled    10.1.2          
      Core                              Comment (comment)                                                                    Disabled   10.1.2          
      Core                              Configuration Manager (config)                                                       Enabled    10.1.2          
      Multilingual                      Configuration Translation (config_translation)                                       Enabled    10.1.2          
      Core                              Contact (contact)                                                                    Disabled   10.1.2          
      Core                              Content Moderation (content_moderation)                                              Disabled   10.1.2          
      Multilingual                      Content Translation (content_translation)                                            Enabled    10.1.2          
      Core                              Contextual Links (contextual)                                                        Enabled    10.1.2          
      Field types                       Datetime (datetime)                                                                  Enabled    10.1.2          
      Field types                       Datetime Range (datetime_range)                                                      Enabled    10.1.2          
      Core                              Database Logging (dblog)                                                             Enabled    10.1.2          
      Core                              Internal Dynamic Page Cache (dynamic_page_cache)                                     Enabled    10.1.2          
      Core                              Text Editor (editor)                                                                 Enabled    10.1.2          
      Core                              Field (field)                                                                        Enabled    10.1.2          
      Core (Experimental)               Field Layout (field_layout)                                                          Disabled   10.1.2          
      Core                              Field UI (field_ui)                                                                  Enabled    10.1.2          
      Field types                       File (file)                                                                          Enabled    10.1.2          
      Core                              Filter (filter)                                                                      Enabled    10.1.2          
      Core                              Forum (forum)                                                                        Disabled   10.1.2          
      Core                              Help (help)                                                                          Disabled   10.1.2          
      Core (Experimental)               Help Topics (help_topics)                                                            Disabled   10.1.2          
      Core                              History (history)                                                                    Enabled    10.1.2          
      Field types                       Image (image)                                                                        Enabled    10.1.2          
      Core                              Inline Form Errors (inline_form_errors)                                              Disabled   10.1.2          
      Web services                      JSON:API (jsonapi)                                                                   Disabled   10.1.2          
      Multilingual                      Language (language)                                                                  Enabled    10.1.2          
      Core                              Layout Builder (layout_builder)                                                      Disabled   10.1.2          
      Core                              Layout Discovery (layout_discovery)                                                  Disabled   10.1.2          
      Field types                       Link (link)                                                                          Enabled    10.1.2          
      Multilingual                      Interface Translation (locale)                                                       Enabled    10.1.2          
      Core                              Media (media)                                                                        Enabled    10.1.2          
      Core                              Media Library (media_library)                                                        Enabled    10.1.2          
      Core                              Custom Menu Links (menu_link_content)                                                Enabled    10.1.2          
      Core                              Menu UI (menu_ui)                                                                    Enabled    10.1.2          
      Migration                         Migrate (migrate)                                                                    Disabled   10.1.2          
      Migration                         Migrate Drupal (migrate_drupal)                                                      Disabled   10.1.2          
      Migration                         Migrate Drupal UI (migrate_drupal_ui)                                                Disabled   10.1.2          
      Core                              MySQL (mysql)                                                                        Disabled   10.1.2          
      Core                              Node (node)                                                                          Enabled    10.1.2          
      Field types                       Options (options)                                                                    Enabled    10.1.2          
      Core                              Internal Page Cache (page_cache)                                                     Enabled    10.1.2          
      Core                              Path (path)                                                                          Enabled    10.1.2          
      Core                              Path alias (path_alias)                                                              Enabled    10.1.2          
      Core                              PostgreSQL (pgsql)                                                                   Enabled    10.1.2          
      Core                              Password Compatibility (phpass)                                                      Disabled   10.1.2          
      Core                              Responsive Image (responsive_image)                                                  Disabled   10.1.2          
      Web services                      RESTful Web Services (rest)                                                          Disabled   10.1.2          
      Core (Experimental)               Single Directory Components (sdc)                                                    Disabled   10.1.2          
      Core                              Search (search)                                                                      Enabled    10.1.2          
      Web services                      Serialization (serialization)                                                        Disabled   10.1.2          
      Core                              Settings Tray (settings_tray)                                                        Disabled   10.1.2          
      Core                              Shortcut (shortcut)                                                                  Enabled    10.1.2          
      Core                              SQLite (sqlite)                                                                      Disabled   10.1.2          
      Core                              Statistics (statistics)                                                              Disabled   10.1.2          
      Core                              Syslog (syslog)                                                                      Disabled   10.1.2          
      Core                              System (system)                                                                      Enabled    10.1.2          
      Core                              Taxonomy (taxonomy)                                                                  Enabled    10.1.2          
      Field types                       Telephone (telephone)                                                                Enabled    10.1.2          
      Field types                       Text (text)                                                                          Enabled    10.1.2          
      Core                              Toolbar (toolbar)                                                                    Enabled    10.1.2          
      Core                              Tour (tour)                                                                          Disabled   10.1.2          
      Core                              Activity Tracker (tracker)                                                           Disabled   10.1.2          
      Core                              Update Manager (update)                                                              Enabled    10.1.2          
      Core                              User (user)                                                                          Enabled    10.1.2          
      Core                              Views (views)                                                                        Enabled    10.1.2          
      Core                              Views UI (views_ui)                                                                  Enabled    10.1.2          
      Core                              Workflows (workflows)                                                                Disabled   10.1.2          
      Core (Experimental)               Workspaces (workspaces)                                                              Disabled   10.1.2          
      Development                       Devel (devel)                                                                        Disabled   5.1.2           
      Development                       Devel Generate (devel_generate)                                                      Enabled    5.1.2           
      Feeds                             Feeds (feeds)                                                                        Enabled    8.x-3.0-beta4   
      Feeds                             Feeds Log (feeds_log)                                                                Disabled   8.x-3.0-beta4 

    Maybe it is about your composer configuration?

  • ddev composer require 'drupal/feeds:^3.0@beta'
    ./composer.json has been updated
    Running composer update drupal/feeds
    Gathering patches for root package.
    > DrupalComposerManaged\ComposerScripts::preUpdate
    Loading composer repositories with package information
    Updating dependencies
    Your requirements could not be resolved to an installable set of packages.
    
      Problem 1
        - laminas/laminas-text 2.9.0 requires php ^7.3 || ~8.0.0 || ~8.1.0 -> your php version (8.2.0; overridden via config.platform, actual: 8.2.8) does not satisfy that requirement.
        - drupal/upgrade_status 4.0.0 requires mathieuviossat/arraytotexttable ~1.0.0 -> satisfiable by mathieuviossat/arraytotexttable[v1.0.9].
        - mathieuviossat/arraytotexttable v1.0.9 requires laminas/laminas-text ^2.9 -> satisfiable by laminas/laminas-text[2.9.0].
        - drupal/upgrade_status is locked to version 4.0.0 and an update of this package was not requested.
    
    
    Installation failed, reverting ./composer.json and ./composer.lock to their original content.
  • 🇳🇱Netherlands MegaChriz

    @tonytheferg
    Hm. At first glance that looks related to the dependency "laminas/laminas-servicemanager":

    • laminas/laminas-text v2.9 is not compatible with PHP 8.2 and requires laminas/laminas-servicemanager ^3.4
    • laminas/laminas-text v2.10 is compatible with PHP 8.2 and requires laminas/laminas-servicemanager ^3.19.0
    • laminas/laminas-servicemanager > 3.16 < 4.0 is not compatible with Drupal 10, because that version of laminas/laminas-servicemanager requires psr/container: ^1.0 while Drupal 10 requires psr/container: ^2.0.

    I don't know yet why you get the error while you try to require Feeds. Because laminas/laminas-feed does not require laminas/laminas-text and since v2.21.0 it no longer requires laminas/laminas-servicemanager either.

    It would have made more sense that you weren't able to require drupal/upgrade_status on PHP 8.2. Or does this happen on a Drupal 9 install?

  • 🇩🇰Denmark ressa Copenhagen

    You could try removing Upgrade Status, and see if that helps? 💬 Upgrade status must be uninstalled before upgrading to Drupal 10 Fixed

  • 🇳🇱Netherlands MegaChriz

    @tonytheferg
    Okay, when I try to require drupal/upgrade_status, it picks mathieuviossat/arraytotexttable 1.0.7 which still requires zendframework packages. zendframework packages have been replaced by laminas, so I would call zendframework packages to be "outdated".

    So the possible solutions that I can think of right now are:

    1. Do not use drupal/upgrade_status.
    2. Ask upgrade_status module to find a replacement for mathieuviossat/arraytotexttable, because that package now installs outdated dependencies on PHP 8.2.
    3. Make sure that laminas/laminas-servicemanager gets a stable 4.0 release, which is compatible with Drupal 10. After that, request that laminas/laminas-text is made compatible with laminas/laminas-servicemanager 4.0.
    4. Ask laminas/laminas-text if the dependency on laminas/laminas-servicemanager can be removed. It was removed for laminas/laminas-feed after I requested to add compatibility with laminas/laminas-servicemanager 4.0.
    5. Juggle with settings in your composer.json to allow installation of dependencies that are incompatible on paper.
  • Doh!! I'll try to remove upgrade status and report back

  • That was it:
    Here are the results after removing upgrade status module.

    ddev composer require 'drupal/feeds:^3.0@beta'
    Info from https://repo.packagist.org: #StandWithUkraine
    ./composer.json has been updated
    Running composer update drupal/feeds
    Gathering patches for root package.
    > DrupalComposerManaged\ComposerScripts::preUpdate
    Loading composer repositories with package information
    Updating dependencies
    Lock file operations: 4 installs, 0 updates, 0 removals
      - Locking drupal/feeds (3.0.0-beta4)
      - Locking laminas/laminas-escaper (2.12.0)
      - Locking laminas/laminas-feed (2.21.0)
      - Locking laminas/laminas-stdlib (3.17.0)
    Writing lock file
    Installing dependencies from lock file (including require-dev)
    Package operations: 4 installs, 0 updates, 0 removals
    Gathering patches for root package.
    Gathering patches for dependencies. This might take a minute.
      - Installing laminas/laminas-stdlib (3.17.0): Extracting archive
      - Installing laminas/laminas-escaper (2.12.0): Extracting archive
      - Installing laminas/laminas-feed (2.21.0): Extracting archive
      - Installing drupal/feeds (3.0.0-beta4): Extracting archive
    5 package suggestions were added by new dependencies, use `composer suggest` to see details.
    Generating autoload files
    101 packages you are using are looking for funding.
    Use the `composer fund` command to find out more!
    phpstan/extension-installer: Extensions installed
    Scaffolding files for pantheon-systems/drupal-integrations:
      - Skip [web-root]/sites/default/settings.php because it already exists and overwrite is false.
    > DrupalComposerManaged\ComposerScripts::postUpdate
    No security vulnerability advisories found
  • 🇹🇷Turkey Orkut Murat Yılmaz

    Well, we will be able to use the stable version of feeds 8.3.0 with php8.2, when it's released.

    Thank you @MegaChriz and the others too:)

Production build 0.69.0 2024