Pages are processed multiple times before a full channel import

Created on 4 April 2024, 9 months ago
Updated 15 May 2024, 7 months ago

Problem/Motivation

Each cron execution adds the first page always in the middle of the execution. This then adds the second, third and so on pages all over again before the whole channel is processed.

Steps to reproduce

E.g. with page 1 limit per process:

  1. drush cron
  2. queued page 1
  3. syncing of page 1 creates a queue item for page 2
  4. drush cron after the interval has elapsed
  5. page 1 is added to the queue again
  6. page 2 is being processed and page 3 is queued
  7. drush cron after the interval has elapsed
  8. page 1 is now being processed again since its queue item was created before the one for page 3.

Re-processing the first page will then also create a queue item for page 2, which on its turn will create a queue item for page 3 and so on. In the meantime page 1 would have been queued again and this hugely delays the processing of the whole channel.

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

πŸ› Bug report
Status

Needs review

Version

3.0

Component

Code

Created by

πŸ‡©πŸ‡ͺGermany hchonov πŸ‡ͺπŸ‡ΊπŸ‡©πŸ‡ͺπŸ‡§πŸ‡¬

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024