Contrib.social

Add a garbage collector to clean up the queue for failed sitemap batches

Open on Drupal.org →
Created on 8 February 2023, about 2 years ago

Problem/Motivation

The problem is, I'm working on a website with more than 10k items to be indexed, and the items are all reindexed every day, after some point, the sitemap batch stopped to work properly, due to a custom module that we have here, the main problem was, it took some days to realize this kind of behavior, and after taking a look at the queue table, we noticed the table with more than 100k items, the sitemap was taking a long time to run. Anyway, after truncating the table the slowness problem was solved.

The idea of this issue is to implement a garbage collector for failed items. Drupal already provides this kind of stuff, if you take a look at the DatabaseQueue class , you'll find a garbage collector method, the problem is, this method expects a drupal_batch: pattern on the name of the queue item, but sitemap uses simple_sitemap_elements. Drupal will clean up the queue table with the correct after 10 days.

Proposed resolution

My idea here is to update the hook cron and add the properly clean up here.

We've could also add a config to configure how much do we want to keep the register on the table.

📌 Task
Status

Active

Version

4.0

Component

Code

Created by

🇧🇷Brazil murilohp

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @murilohp
  • Status changed to Needs review about 2 years ago
  • Comment about 2 years ago →
    🇧🇷Brazil murilohp

    Here's a patch with the hook cron updated, I'm moving to NR to see what you guys think.

  • Comment about 2 years ago →
    🇧🇷Brazil murilohp
  • Status changed to Postponed: needs info about 2 years ago
  • Comment about 2 years ago →
    🇩🇪Germany gbyte Berlin

    That's interesting; thanks for the patch! Now I feel I have to check the batch table in my projects. What do other contrib modules do - build their batch garbage collectors like you did, or prefix their batch entries with 'drupal_batch:'?

  • Comment 12 months ago →
    🇳🇱Netherlands jaapjan

    Also using this patch on my project because the queue is full of simple_sitemap_elements. Any plans on implementing a solution in the module?

  • Comment 4 months ago →
    🇷🇺Russia walkingdexter

    Need steps to reproduce. The queue is cleared before rebuilding.

  • Status changed to Closed: works as designed 11 days ago
  • Comment 11 days ago →
    🇷🇺Russia walkingdexter

    Closing due to lack of activity. Feel free to reopen if you can provide more information.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024