The update module has to individually fetch project XML for each project installed on a site. This can reach 300+ http requests which is a lot.
However, guzzle supports async support and promises, so it might be possible to rewrite this using Fibers and guzzle promises. As long as we don't hit memory issues, we might be able to then skip the batch support and do it directly.
There are two components to getting update data for an extension:
1. Fetch the project XML
2. Parse the project XML.
Ideally we'd do something like this:
1. Have the list of projects to fetch update data for.
2. Issue up to x number of async requests for the project xml.
3. As soon as one request comes back, issue another x async http requests, process the html for any that have come back, rinse and repeat until no more requests or XML to process.
Need some sort of absolute cap on outgoing http requests too.
Had a look at doing this 'quickly' but found the http retry logic which makes it a bit more complicated to implement, so just opening the issue for now.