Improve rate limit handling

Created on 24 December 2024, 4 months ago

Problem/Motivation

Doppio has certain rate limits. With our plan, those are 400 requests per month and 20 requests per 60 seconds. Currently, the queue worker keeps on retrying indefinitely when it encounters those kind of rate limits.

Proposed resolution

When encountering a rate limit, delay the queue item until the rate limit has passed.

📌 Task
Status

Active

Version

1.0

Component

Code

Created by

🇧🇪Belgium dieterholvoet Brussels

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @dieterholvoet
  • 🇧🇪Belgium dieterholvoet Brussels

    I changed the error handling for queues, but not sure what to do with the Batch API method. Error handling used to be that errors are logged but end users aren't notified. Do you still think this is the way to go?

  • Merge request !25Improve rate limit handling → (Open) created by dieterholvoet
  • Pipeline finished with Success
    4 months ago
    Total: 169s
    #377797
  • 🇬🇧United Kingdom scott_euser

    Hmmm we've never used it at scale in bulk so never had the need yet, but really good feature! Seems like better to show the errors to the user then if run via batch. A rate limit error would hint to them to manually try again later mimicking the automated delay of queue worker.

    Looking at your MR at a glance the delay might be a month? If so I wonder if you'd get support requests from your clients asking why it's not generated yet. Would it make sense to add a messenger status to node edit like 'PDF generation is queued for this content item to be processed after (date time)'?

Production build 0.71.5 2024