429 Too Many Requests

On further investigation, this might be an issue with Migrate Tools module. When running through the API, only a single item is imported per batch request. But running through Drush seems to circumvent the problem. Note when running through Drush, When using Drush, only 2 HTTP requests are made total.

Having now spent a couple of weeks regularly running into this error of "Too many requests" from Google's API, and having failed to request a rate limit increase from Google themselves, I took a hand at resolving this issue by extending this module's plugin class. Having put it together, I think solving the problem here through a configuration option would be the best solution.

I would suggest that a YAML configuration option be added like so:

  # Specify a cache lifetime in seconds. This will persist a local cache of
  # remote spreadsheet API request responses. This can be used to lower
  # the number of API requests to fit within rate limits and avoid a
  # "429 Too Many Requests" response from Google. Omit or set to 0 to
  # disable the cache.
  cache_lifetime: 10

Do you have the cache lifetime implemented in your plugin? If you open a MR I can take a look, I've seen the same myself when hitting through the UI but so far no issues just running with Drush.

I've seen the same myself when hitting through the UI but so far no issues just running with Drush.

Yeah this issue crops up very easily in the UI. For drush, you need to have a lot of migrations (20+) running via Drush for it to hit the limit, but we ran into that a fair amount also.

Sorry this fell off my radar since we got it working for our project, but we may be revisiting this soon.

Sure! The project I'm using this on has 17 migrations so I'm not hitting rate limits, but I can take a look at whatever you've got!

In the MR I changed getSourceData() to more closely match what the parent Json::getSourceData() is doing, but it looks like executing it through the UI is where the issues occur.

When executing with Drush I'm only seeing two HTTP requests with or without these changes.

Adding two related issues about caching, in #2826938: Integrate HTTP fetcher with Guzzle Cache if available → heddn suggests using the guzzle_cache project, but that would rely on cache headers and it looks like the Sheets API is returning Cache-Control: max-age=0

Caching is opt-in, cache_lifetime: n needs to be added to the source configuration.