Retry-After header has 0 value which breaks common retry strategies

PHP Fatal error: Uncaught GuzzleHttp\Exception\ClientException: Client error: `GET https://www.drupal.org/api-d7/node.json?type%5B0%5D=project_module&type%5B1%5D=project_theme&type%5B2%5D=project_core&field_project_has_releases=1&field_project_type=full&sort=field_project_machine_name&page=119` resulted in a `429 Too Many Requests` response: { "Connection": [ "close" ], "Content-Length": [ "164" ], "Server": [ "Varnish" ], "Retry-After": [ "0" ], "Content-Type": [ "text\/html" ], "Accept-Ranges": [ "bytes" ], "Date": [ "Thu, 03 Apr 2025 08:00:15 GMT" ], "Via": [ "1.1 varnish" ], "Set-Cookie": [ "_pxhd=oT4P4j2loDaMQqrpLz9dGwvplwHr1Hfy4h5F7BldKJy\/pGV17\/icGbFG9tZZ-qXZYpbikfPmwypRuPOvFbZrTQ==:J87ivGXXLvIn55oxCBAJu207lwOc\/IH38bDTOcCww9VuYBrZvFogAjAcNEtmwO74HW9S1XwI-3cndcwvypezMHPdocOyCX1\/reXMV9lLOPU=; Expires=Fri, 03 Apr 2026 08:00:15 GMT; path=\/;" ], "X-Served-By": [ "cache-pao-kpao1770059-PAO" ], "X-Cache": [ "MISS" ], "X-Cache-Hits": [ "0" ], "X-Kevinrob-Cache": [ "MISS" ] } <html> <head> <title>Too Many Requests</title> </head> <body> <h1>Too Many Requests</h1> <p>Reached maximum requests limitation, try again soon.</p> </body> </html> in /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 Stack trace: #0 /home/runner/work/ddqg/ddqg/src/Infrastructure/DrupalOrg/DrupalOrgApi/CustomErrorHandler.php(40): GuzzleHttp\Exception\RequestException::create() #1 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(209): mxr576\ddqg\Infrastructure\DrupalOrg\DrupalOrgApi\CustomErrorHandler::mxr576\ddqg\Infrastructure\DrupalOrg\DrupalOrgApi\{closure}() #2 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(158): GuzzleHttp\Promise\Promise::callHandler() #3 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/TaskQueue.php(52): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}() #4 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(163): GuzzleHttp\Promise\TaskQueue->run() #5 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(189): GuzzleHttp\Handler\CurlMultiHandler->tick() #6 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Handler\CurlMultiHandler->execute() #7 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn() #8 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending() #9 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList() #10 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending() #11 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/EachPromise.php(109): GuzzleHttp\Promise\Promise->wait() #12 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Promise\EachPromise->GuzzleHttp\Promise\{closure}() #13 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn() #14 /home/runner/work/ddqg/ddqg/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending() #15 /home/runner/work/ddqg/ddqg/src/Infrastructure/DrupalOrg/DrupalOrgApi/DrupalOrgApiRepository.php(192): GuzzleHttp\Promise\Promise->wait() #16 /home/runner/work/ddqg/ddqg/src/Infrastructure/DrupalOrg/DrupalOrgApi/DrupalOrgApiRepository.php(71): mxr576\ddqg\Infrastructure\DrupalOrg\DrupalOrgApi\DrupalOrgApiRepository->fetchProjectNames() #17 /home/runner/work/ddqg/ddqg/bin/ddqg-generate-no-unsupported-versions(26): mxr576\ddqg\Infrastructure\DrupalOrg\DrupalOrgApi\DrupalOrgApiRepository->fetchProjectIds() #18 {main} Next RuntimeException: Failed to fetch page 119. Reason: "Client error: `GET https://www.drupal.org/api-d7/node.json?type%5B0%5D=project_module&type%5B1%5D=project_theme&type%5B2%5D=project_core&field_project_has_releases=1&field_project_type=full&sort=field_project_machine_name&page=119` resulted in a `429 Too Many Requests` response: { "Connection": [ "close" ], "Content-Length": [ "164" ], "Server": [ "Varnish" ], "Retry-After": [ "0" ], "Content-Type": [ "text\/html" ], "Accept-Ranges": [ "bytes" ], "Date": [ "Thu, 03 Apr 2025 08:00:15 GMT" ], "Via": [ "1.1 varnish" ], "Set-Cookie": [ "_pxhd=oT4P4j2loDaMQqrpLz9dGwvplwHr1Hfy4h5F7BldKJy\/pGV17\/icGbFG9tZZ-qXZYpbikfPmwypRuPOvFbZrTQ==:J87ivGXXLvIn55oxCBAJu207lwOc\/IH38bDTOcCww9VuYBrZvFogAjAcNEtmwO74HW9S1XwI-3cndcwvypezMHPdocOyCX1\/reXMV9lLOPU=; Expires=Fri, 03 Apr 2026 08:00:15 GMT; path=\/;" ], "X-Served-By": [ "cache-pao-kpao1770059-PAO" ], "X-Cache": [ "MISS" ], "X-Cache-Hits": [ "0" ], "X-Kevinrob-Cache": [ "MISS" ] } <html> <head> <title>Too Many Requests</title> </head> <body> <h1>Too Many Requests</h1> <p>Reached maximum requests limitation, try again soon.</p> </body> </html> ". in /home/runner/work/ddqg/ddqg/src/Infrastructure/DrupalOrg/DrupalOrgApi/DrupalOrgApiRepository.php:187

Comments & Activities

Issue created by @mxr576
Comment 4 days ago →
🇺🇸United States drumm NY, US
We recently enabled this new-to-us feature from a service provider, in response to API requests sent at an abusive rate. It looks like this is something we can customize to correct.

Our initial setting for the rate limit was somewhat arbitrary. Do you have a sense of how far below your previous crawl rate this kicked in at?
Comment 4 days ago →
🇺🇸United States drumm NY, US
retry-after: 10 is now in the response headers.
Comment 4 days ago →
🇭🇺Hungary mxr576 Hungary
Do you have a sense of how far below your previous crawl rate this kicked in at?

My script uses concurrency 10 with Guzzle.

Thanks for fixing it so quickly! I had a workaround in place, will leave it for a time being.

Retry-After header has 0 value which breaks common retry strategies

Problem/Motivation

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Comments & Activities