Gracefully handle 429 Too Many Requests

Created on 23 May 2023, over 1 year ago
Updated 18 July 2023, over 1 year ago

Problem/Motivation

If there are too many requests the stackpath api service will just give up and log an error. However it would be nice if it would retry the request at least once before failing.

We've got the issue while running these two updates that are installing and uninstalling fields and therefore :

drush updatedb -vvv -y
drush locale-check
drush locale-update

  Module   Update ID   Type            Description                                          
 -------- ----------- --------------- ----------------------------------------------------- 
  co       8206        hook_update_n   8206 - Add created and changed fields to stores.     
  mm                                                                                        
  er                                                                                        
  ce                                                                                        
  _s                                                                                        
  to                                                                                        
  re                                                                                        
  co       8207        hook_update_n   8207 - Uninstall the created / changed fields added  
  mm                                   by content_translation.                              
  er                           ...l enthรคlt Details.
> 
>  [notice] Message: 16 รœbersetzungsdateien wurden importiert. /0/ รœbersetzungen wurden 
> hinzugefรผgt, /20473/ รœbersetzungen wurden aktualisiert und /0/ 
> รœbersetzungen wurden entfernt.
> 
>  [notice] Message: Es wurden keine Konfigurationsobjekte aktualisiert.
> 
>  [error]  Client error: `POST https://gateway.stackpath.com/cdn/v1/stacks/schwabe-pharma-7ea11d/purge` resulted in a `429 Too Many Requests` response:
> {"code":8,"message":"Too many CDN purge requests received. Please retry this request in 31 seconds.","details":[{"@type" (truncated...)
>  
 [error]  Client error: `POST https://gateway.stackpath.com/cdn/v1/stacks/schwabe-pharma-7ea11d/purge` resulted in a `429 Too Many Requests` response:
{"code":8,"message":"Too many CDN purge requests received. Please retry this request in 28 seconds.","details":[{"@type" (truncated...)

Steps to reproduce

Execute too many requests in a short amount of time.

Proposed resolution

Either on 429 get the seconds to wait from the response, wait that much time and retry the request once or find out why some updates result into so many invalidations and try to invalidate only once at the end of the deployment.

Remaining tasks

User interface changes

API changes

Data model changes

โœจ Feature request
Status

Fixed

Version

1.0

Component

Code

Created by

๐Ÿ‡ฉ๐Ÿ‡ชGermany hchonov ๐Ÿ‡ช๐Ÿ‡บ๐Ÿ‡ฉ๐Ÿ‡ช๐Ÿ‡ง๐Ÿ‡ฌ

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @hchonov
  • Status changed to Needs review over 1 year ago
  • Open on Drupal.org โ†’
    Core: 10.0.7 + Environment: PHP 7.4 & MySQL 5.7
    last update over 1 year ago
    Waiting for branch to pass
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany hchonov ๐Ÿ‡ช๐Ÿ‡บ๐Ÿ‡ฉ๐Ÿ‡ช๐Ÿ‡ง๐Ÿ‡ฌ
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany hchonov ๐Ÿ‡ช๐Ÿ‡บ๐Ÿ‡ฉ๐Ÿ‡ช๐Ÿ‡ง๐Ÿ‡ฌ
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany hchonov ๐Ÿ‡ช๐Ÿ‡บ๐Ÿ‡ฉ๐Ÿ‡ช๐Ÿ‡ง๐Ÿ‡ฌ
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany a.dmitriiev

    I guess it is better to check the exception status code, like $e->getCode() == 429 instead of doing regular expression match on the message, this will be a bit easier.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany a.dmitriiev

    And I think it is better to re-queue the items instead of do sleep inside this method from patch. Check queuer plugin, e.g. this one from purge module purge/modules/purge_queuer_coretags/src/Plugin/Purge/Queuer/CoreTagsQueuer.php, it is used in this service purge/modules/purge_queuer_coretags/src/CacheTagsQueuer.php. I think it is better to create some RetryQueuer plugin and just add failed invalidations back to the queue

  • Status changed to Needs work over 1 year ago
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany hchonov ๐Ÿ‡ช๐Ÿ‡บ๐Ÿ‡ฉ๐Ÿ‡ช๐Ÿ‡ง๐Ÿ‡ฌ

    I guess it is better to check the exception status code, like $e->getCode() == 429 instead of doing regular expression match on the message, this will be a bit easier.

    oh, I was thinking to add the check for the code, so it looks like I've missed this. But checking the error message is needed to get the seconds needed to wait.

    And I think it is better to re-queue the items instead of do sleep inside this method from patch. Check queuer plugin, e.g. this one from purge module purge/modules/purge_queuer_coretags/src/Plugin/Purge/Queuer/CoreTagsQueuer.php, it is used in this service purge/modules/purge_queuer_coretags/src/CacheTagsQueuer.php. I think it is better to create some RetryQueuer plugin and just add failed invalidations back to the queue

    Of course, if it was using a queue, which is not the case if the API is invoked directly through a drush command.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany a.dmitriiev

    All purging is done through the queue even the one that is invoked from drush command. I assume the fix from this patch should be enough. The queue item will be set to Failed, and according to QueueService "Failed" items go back to the queue automatically.

  • Status changed to Needs review over 1 year ago
  • Open on Drupal.org โ†’
    Core: 10.0.7 + Environment: PHP 7.4 & MySQL 5.7
    last update over 1 year ago
    Waiting for branch to pass
  • Open on Drupal.org โ†’
    Core: 10.0.7 + Environment: PHP 7.4 & MySQL 5.7
    last update over 1 year ago
    Waiting for branch to pass
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany hchonov ๐Ÿ‡ช๐Ÿ‡บ๐Ÿ‡ฉ๐Ÿ‡ช๐Ÿ‡ง๐Ÿ‡ฌ

    Just re-rolling.

  • Status changed to Fixed over 1 year ago
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024