Add a setting for limiting the amount of times a lock may be extended

Created on 14 November 2024, 2 months ago

Problem/Motivation

Coming from 🐛 Feed fetch errors don't seem to be handled gracefully Active .

When an import starts, the feed gets locked for 12 hours. When the lock expires, Feeds checks if the import is perhaps still running by checking:

  • If there are still tasks for the feed on the queue (relevant for cron imports).
  • If there was progress being made in the last hour (more relevant for manual imports).

Only one of the above has to be true for the lock getting extended.

However, when an import gets stuck for whatever reason, Feeds may retry continue the import forever. In this scenario, when a task from the queue errors, the task remains on the queue, causing Feeds to think that the import is still active.

To get out of the loop, there should be a configurable limit for how many times the lock may be extended. With the default that is not extended. The only downside to this default is that people running imports that take longer than 12 hours to complete need to be pay extra attention to this setting, but the benefit is that there is by default a limit on how long imports can be a stuck state.

Steps to reproduce

Proposed resolution

On the feed type settings, we could have the following setting:

Keep the feed locked if the import has not completed after the lock timeout

The lock timeout is: 12 hours (taken from feeds.settings.lock_timeout).

  • Never (default)
  • Always
  • Extend the lock this many times at max: [number field]

In FeedLockBackend::lockMayBeAvailable(), the setting should be checked for if it lock may be extended or not. If the setting is set to a number, it would need check a state variable to see how many times the lock for this feed was already extended. Then it need to plus that number and write that back to the state variable.
I think it makes sense to get the state this way, though it does require making the method getStateStorage() public:

$lock_extended_count = $feed->getStateStorage()->get('lock_extended_count');

Remaining tasks

  • On the feed type, add a setting for the lock extension. Should be added to FeedTypeForm::form().
  • Make Feed::getStateStorage() public.
  • Add the getStateStorage() method to FeedInterface.
  • In FeedLockBackend, check if the lock may be extended. If it is an exact number of times, use the state variable "lock_extended_count"
  • Add test coverage to Drupal\Tests\feeds\Kernel\Lock\FeedsLockBackendTest:
    • Add a test that checks if the lock does not get extended if the setting is set to 'Never'.
    • Add a test that checks if the lock only gets extended once if it is set to '1'.
    • Add a test that checks if the lock only gets extended twice if it is set to '2'.
    • Add a test that checks if the lock keeps getting extended if it is set to 'Always'. Maybe just trigger 5 lock extensions?

User interface changes

API changes

Data model changes

Feature request
Status

Active

Version

3.0

Component

Code

Created by

🇳🇱Netherlands megachriz

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024