First page is processed multiple times

Created on 4 April 2024, almost 2 years ago

Problem/Motivation

Each cron execution adds the first page always in the middle of the execution.

Steps to reproduce

E.g. with page 1 limit per process:

drush cron
queued page 1
syncing of page 1 creates a queue item for page 2
drush cron after the interval has elapsed
page 1 is added to the queue again
page 2 is being processed and page 3 is queued
drush cron after the interval has elapsed
page 1 is now being processed again since its queue item was created before the one for page 3.

All this unnecessarily syncs page 1 and also slows down the whole channel sync. Processing the first page will then also create a queue item for page 2 again further delaying the processing of later pages.

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

🐛 Bug report

Status

Needs work

Version

3.0

Component

Code

Created by

🇩🇪Germany hchonov 🇪🇺🇩🇪🇧🇬

Live updates comments and jobs are added and updated live.

Comments & Activities

Issue created by @hchonov
Comment almost 2 years ago →
🇩🇪Germany hchonov 🇪🇺🇩🇪🇧🇬
Status changed to Needs review almost 2 years ago2:14pm 5 April 2024
Comment almost 2 years ago →
🇩🇪Germany hchonov 🇪🇺🇩🇪🇧🇬
Comment over 1 year ago →
🇩🇪Germany hchonov 🇪🇺🇩🇪🇧🇬
I've noticed there were still a lot of duplicates in the queue, so adding a condition to not create a queue item if the same one already exists in the queue.

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024