Large feeds do not import entirely due to missing temp file

Created on 18 October 2024, 3 months ago

Problem/Motivation

Using this module, the feed import processes and completes, but does not import the entire feed. The problem only occurs on large feeds that take multiple cron jobs. Testing reproduced issue on load-balanced environments (Pantheon). The following error message is produced:
The feed "XX" encountered an error when processing: Drupal\feeds\Exception\FileNotFoundException: File <em class="placeholder">/tmp/feeds_http_fetcherXxx</em> does not exist. in Drupal\feeds\Result\FetcherResult->checkFile() (line 58 of /code/web/modules/contrib/feeds/src/Result/FetcherResult.php).
Downloading the temp file from the Feed log (or sftp) shows the temp file contains the entire feed.

The issue has been narrowed down to this module due to it setting the temporary file location.

Steps to reproduce

  1. Use a Drupal site on a load-balanced environment.
  2. Start import (in background) of a feed that uses feeds_fetcher_post with over 1000 entries.
  3. Wait for cron to run a few times to complete import.
  4. Check feed log at /feed/XX/log to see how many entries were imported.
  5. Feed log shows less than 1000 entries imported, usually a number such as 300, 700 or 900.
  6. DBLog at /admin/reports/dblog?type%5B%5D=feeds shows error as above.
  7. Feed temp file can be downloaded and is viewable through sftp from the private://feeds/log/XXXXX/source/ location created by feeds module, but not from the /tmp/ location created by the feeds_fetcher_post module.

Proposed resolution

If feeds_fetcher_post would not override the temp file location of HttpFetcher in feeds module, then this issue can be fixed.

Remaining tasks

User interface changes

API changes

Data model changes

πŸ› Bug report
Status

Active

Version

1.2

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States andreayaya

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Production build 0.71.5 2024