Contrib.social

Processing of failed queues never finish, feeds import gets stuck

Open on Drupal.org →
Created on 26 March 2019, about 5 years ago
Updated 11 April 2023, about 1 year ago

Problem/Motivation

This issue happens, when you have configured feeds import by cron.

Sometimes queue gets stuck. There could be different reasons, for example tmp folder was cleared and temporary xml file was removed, but queue was not finished yet or due timeout error on external server.
When queue fails, it throws exception error in FeedsRefresh->processItem(). So this queue will never finish. This also influence on further imports for this specific feed. For example if next time new queue will be created, the old one (failed) still exist in db and will never deleted. This interrupts updating of imported items, but status page shows that feed was updated. In fact items still has outdated data. If you run import manually it updates items properly, but it not fixes the problem for import via cron.

Proposed resolution

  1. Need to delete queues, which stuck due some error (during queue processing or by cron, but lifetime should be configurable).
  2. Need to add more monitoring information for feeds and ability to reset each feed
    • By monitoring I mean, that on feeds overview page (/admin/content/feed) would be nice to show queued time, how many queues currently processing by cron.
    • There should be action link for reset feed: delete all existing queues related to this feed, reset queued time, run import again like it will be run via cron
🐛 Bug report
Status

Active

Version

3.0

Component

Code

Created by

🇺🇦Ukraine maxdev

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment about 1 year ago →
    🇮🇳India ShubhamJ

    @maxdev

    How can we programatically set queue to 0?
    Also, should we replace Queue UI with other module?
    Can you suggest?

  • Comment about 1 year ago →
    🇺🇸United States bogdog400

    I've had this happen lately with a very long CSV file filled with media URLs. The percentage completed oscillates. It runs up to 90% and then drops back down to 10%. It's confusing.

  • Comment about 1 year ago →
    🇺🇸United States fredonia_webteam

    We have experienced this issue with 825 items, which updates every 3 hours. The import stalls at different points in the process. We have included screen captures for you to view:

    • log.png - shows the log of the import. Each stall shows the import at various counts (200, 400, 600). Note the import works after unlocking and then running drush feeds:import (825 items).
    • status.png - show the status of the import. The last update (from this morning) was 1:00am, with the import start time of yesterday at 9:21am.

    We have other (short) feeds importing with no issues. Cron runs every hour. We have bumped up the cron frequency, but it does not make any difference. It's as if the other imports overtake the cron, which in turn prevents the large feed from importing. There are no errors on the feed, nor does it ever unlock on its own to allow a fresh import to start. This issue happens in both a load balanced environment and non-load balanced environment.

    We hope these details help with resolving this issue.
    Drupal 9.5.7, Feeds 8.x-3.0-beta3

contrib.social Blog FAQ Discussions
Production build 0.69.0 2024