Problem/Motivation
Trying to do a (json) export during a hook_cron. We can often have tens of thousands of webform submissions for certain forms. We've done programmatic interactive/batch-based exports plenty of times and we know to keep the per-batch-step processing limited to prevent time-outs. This time is different as it's run on cron.
I've checked all subsequent release notes for anything on this, and even checked the latest 6.2 dev source code - it remains the same.
Using Drupal 9.5.10.
Steps to reproduce
Create a number of submissions exceeding the batchLimit, then set (programmatic) export options (like date, etc. - no files or attachments needed) to filter options such that `getTotal()` will result in less than it. Export will fail, as `requiresBatch()` still evals to TRUE. Setting the `limit` option to below `batchLimit` doesn't help. `requiresBatch()` still evals to TRUE even if 0 submissions match. Looking at all the options I don't see anything else that obviously would help.
Discovery
It's related to this change https://www.drupal.org/files/issues/2019-08-06/3069358-15.patch because of the way `$total` is calculated (which does not match what `getTotal()` returns).
that change was:
public function requiresBatch() {
- return ($this->getTotal() > $this->getBatchLimit()) ? TRUE : FALSE;
+ // Get the unfiltered total number of submissions for the webform and
+ // source entity.
+ $total = $this->entityStorage->getTotal(
+ $this->getWebform(),
+ $this->getSourceEntity()
+ );
+ return ($total > $this->getBatchLimit()) ? TRUE : FALSE;
}
the old way seems more intuitive, as I could not understand the reasoning that led to the change of using "unfiltered total number".
Proposed resolution
Dunno... fix/refine the `requiresBatch()` logic, or provide a Queues based "API" that will support exporting during cron?
Remaining tasks
User interface changes
API changes
Data model changes