Huge sitemap

Problem/Motivation

We are implementing sitemap generation for a pretty big project with millions of content entities, so, the process takes a way more time then it would be thousands or even hundreds of thousands items. All data needed for sitemap generation is already in Elasticsearch index. Besides we want to minimize the SQL access as much as possible because it is the bottleneck.
Now let's say we are creating a plugin to generate sitemap using data I mentioned and we have to put something into the queue.
We cannot create a queue item for each entity because it's too many. Also we cannot use queue items as "range/pagination" (there is a limitation in Elasticsearch to 10k items (from+size), so we can only store a "search_after" token). On the other hand we should put something into a queue to initiate processing.
Also we need to generate one specific sitemap (like in example in other issue 💬 Add the ability to process the queue without triggering a rebuild of the queue, even if the queue is empty Active ), currently not possible.

What's the best approach to handle it? Any ideas or suggestions are welcome.

At this point, our ideas for solving the issue are based on customizations in parts of the module that, unfortunately, are not extendable.
But we'd like to contribute back to this module.

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes