Add an 'instant' queue runner

We don't need to use 1px gifs, between these two we can issue http requests from within the request out to the Drupal site then bail without checking the result, which takes around 1ms.

https://github.com/biznickman/PHP-Async

http://drupal.org/project/httprl

Related: #1599622: Run poormanscron via a non-blocking HTTP request →

I don't think that is a problem at all, the issue linked above in fact already adds that plugin.

So I've started having a think about this, but for the async in Guzzle we'll need to add the Async plugin to core (http://guzzlephp.org/guide/plugins.html#async-plugin) - will that be a problem, and is too late in the cycle to get this in before feature freeze?

Was also thinking this queue runner would need to be able to be disabled (in the performance section or elsewhere?) as some use cases may want to prevent anything running "randomly"?

For implementation, I was thinking of extending hook_cron_queue_info() to add a flag for whether the queue specified should be executed by this queue runner. There is hook_queue_info which has been exposed by the contrib Queue UI module (http://drupal.org/project/queue_ui) but I guess getting that into core is pushing it?

Ok great, so for working on it I'll apply the patch from #1599622: Run poormanscron via a non-blocking HTTP request → and we can make that a dependency.

I think I was editing my post as you replied, do you think identifying the queues to be run in queue runner should be configured in hook_cron_queue_info()?

Drupal 8.0.6 → was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 → is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

One further thought on this, why would we only want to spawn these queue runner processes from a form submit? Looking at how the poormanscron will work (and works in D7) we'll be spawning a request to /cron/%process every time we reach the cron processing threshold, is there any reason why we wouldn't want to do the same for queues?

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

Drupal 8.2.0-beta1 → was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

Tried updating the issue summary.

I think the first thing to do here would be to implement something which just helps to run-off a few queue items in-between cron jobs, for #2878119: Whether queued or not, update the media thumbnail and metadata before beginning the entity save database transaction → and similar issues.

So:

- Add an item to the queue that we want processing asap

- Add a session flag with the queue name we want to run off.

- Each request, look for that session flag, and if it's there, render a 1px gif

- The 1px gif links to a URL with the queue name and a token. Configuration/settings sets the number of items to process (default to 5 or 10?) or do it a bit like batch where we do as many as we can within a second or two.

@catch
Do you think this should be a module, just like the automatic_cron module?

@dawehner a new module sounds good to me.

Why not use a dedicated service that will be triggered on post-send request. User will have the response already sent, Drupal will start processing the queue that was just added (almost in sync). This will allow this system to be useful in decoupled scenarios, not only when we render markup and 1px .GIF files.

Drupal 9.3.0-rc1 → was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule → and the Allowed changes during the Drupal core release cycle → .

Drupal 9.2.0-alpha1 → will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule → and the Allowed changes during the Drupal core release cycle → .

Drupal 9.1.0-alpha1 → will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule → and the Allowed changes during the Drupal 9 release cycle → .

Drupal 8.9.0-beta1 → was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule → and the Allowed changes during the Drupal 8 and 9 release cycles → .

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule → and the Allowed changes during the Drupal 8 and 9 release cycles → .

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule → and the Allowed changes during the Drupal 8 release cycle → .

iirc that only works on fastcgi, see #2851111: fastcgi_finish_request and shutdown functions (e.g. batch) do not play nicely together → and similar issues.

Given the queue API can't provide feedback, we can skip the browser altogether now that Fibers is available: 🌱 [meta] Use Fibers for concurrency in core Active

Can a fiber live longer than the main thread or whatever you want to call it? That's why we use that terminate stuff that can run after the response has been sent and the connection closed. If this blocks and keeps the connection open and therefore not sending out the response then that seems problematic and unpredictable?

It can't, and I got overexcited opening this issue before fully reading through the docs. If we were to try to use it for this, it would end up needing to be something like Guzzle's non-blocking HTTP request (which might get refactored on top of a Fiber). Fibers don't allow for simultaneous execution by themselves.

There's already working solution, usable for 9.4 too via react PHP https://mglaman.dev/blog/using-reactphp-run-drupal-tasks

Nowadays both amp and react php works with symfony microkernel https://uvinum.engineering/how-we-are-adding-async-php-to-our-stack-3bb7...

@andypost amp or reactphp might help on the cli, but this requires running a daemon waiting for queue items to come in, which isn't going to be an option on a lot of hosting environments. That's more what https://www.drupal.org/project/advancedqueue → and similar are trying to do.

What I am thinking about here is a module we can ship with core, similar to automated_cron, that will run some queues down in the browser. It is not as good as a waiting queue, but it would allow us to do things like add functionality like https://www.drupal.org/project/image_style_warmer → to core.

When we originally added automated cron to core, one of the ideas was to add a 1px gif following POST requests. This was rejected at the time because it was not guaranteed to run often enough on a site without much authenticated activity, but I think it would be enough here since we're explicitly hoping to execute queue items created by things like saving an entity form.

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened → , as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule → and the Allowed changes during the Drupal core release cycle → .

After 🐛 Post-response task running (destructable services) are actually blocking; add test coverage and warn for common misconfiguration Fixed there's a possible path forwards here. I think we could add it to automated_cron

1. Automated cron adds a decorator for the queue service, this keeps a record of any queue that has an item added during a request.

2. That same decorator implements DestructableInterface and runs some (configurable via container parameter?) number of queue items from the list of queues it has at the end of the request - so could do do 1 item each from five queues or 5 items from one queue or similar.

We cant guarantee that the queue items that were added will be the ones that were created during the request, but some queue items will be processed and anything that doesn't gets picked up by the next cron run.

This would allow us to fix long-standing issues like #504012: Index content when created → , that issue could continue to mark the node for reindex, but also add a queue item that will reindex it when it runs, then automated cron picks that queue item up and reindexes it at the end of the request.

It would also open up the possibility to bring the functionality of https://www.drupal.org/project/image_style_warmer → and similar modules into core - we just create the queue item and if you have automated cron it might handle it, but if also it would work if you have a dedicated permanent queue runner or frequent drush jobs just for running queues too.

I'm proposing the following implementation in automated_cron module
- Instant queue processor will have the following configurations
- Min number of items to trigger the queue. Setting to zero will disable instant queue processing
- Max number of concurrent queue processing
- Implement a queue.database manager extending the core queue.database which will track the items being added to the queue. All queuing function will be delegated to the core implementation.
- Add a new route with no access check that will process the queue
- At end of a request if the number of queue items configured is reached then an async queue processing will be triggered using the mechanism in https://github.com/biznickman/PHP-Async. I've been using this approach and works quite well.

This will not use any browser based implementation or 1x1 gifs.

Any thoughts, suggestions, edits or objections? I'll try to implement the same shortly.

@sukr_s when we originally implemented automated cron in core, we looked at making asynchronous http requests to a Drupal route, but eventually gave up on it because some hosting environments do not cleanly support making http requests to themselves. This is why automated_cron uses the terminate event, to run after the response has been sent to the browser but before script termination. The post response logic works a lot more consistently after 🐛 Post-response task running (destructable services) are actually blocking; add test coverage and warn for common misconfiguration Fixed .

So I would just directly process the queue items in the terminate event listener without trying to deal with async http requests. This also avoid having to worry about an access-free route for processing the queue items, which we'd probably have to lock down with a token or similar.

- Min number of items to trigger the queue. Setting to zero will disable instant queue processing
- Max number of concurrent queue processing

Why not a maximum number of queue items to process, and if there's one or more queue item added during the request, process them at the end? If the maximum number is 0, nothing ever gets processed. I don't see what a minimum number of items gets us.

@catch

So I would just directly process the queue items in the terminate event listener without trying to deal with async http requests

With this suggestion, I'm assuming that processing in the event listener will be non-blocking, otherwise async call is better. Will go with your suggestion for now.

Why not a maximum number of queue items to process,

Do you mean not to process more than the maximum number of items in the queue at a time. I was thinking of setting the time limit to zero so that all the items in the queue will be processed. Otherwise in each terminate event we will have to check the remaining number of items in the queue and trigger again, which would mean additional db calls in the terminate event.

and if there's one or more queue item added during the request, process them at the end?

Yes that's the current thought as well that the process queue call would be done in the terminate event to avoid multiple calls in cases where multiple items are added to the queue.

I don't see what a minimum number of items gets us.

If there was a need to process in a batch instead of immediate processing, perhaps an unwanted frills.

With this suggestion, I'm assuming that processing in the event listener will be non-blocking, otherwise async call is better. Will go with your suggestion for now.

Yes it's non-blocking - the terminate event runs after the response is sent to the browser. Prior to Drupal 10.2-ish this only worked on certain server configurations but since the issue linked above it works on nearly all.

Do you mean not to process more than the maximum number of items in the queue at a time. I was thinking of setting the time limit to zero so that all the items in the queue will be processed. Otherwise in each terminate event we will have to check the remaining number of items in the queue and trigger again, which would mean additional db calls in the terminate event.

The queue could potentially have thousands of items in it, that could lead to OOM errors etc if we tried to process everything. The way I thought of this working was that we would track which queue and how many items are added during a request, probably in a class property of the decorator or similar.

Say someone submits a node form, and that adds one queue item to two different queues, A and B, then in the terminate event, we'd process one item from queue A and one item from queue B. Obviously there's no guarantee that these are the same queue items at all, but this feature is mostly intended for lower traffic sites where that's more likely to be the case.

However if submitting the node form added 500 items to the queue (e.g. it adds queue item for every person following an issue or something), we'd only process the configured maximum of items then stop (anything else would have to eventually be picked up by cron, which also processes queue items).

Left some comments/questions on the MR.

Addressed MR Comments. Moved common code to a trait as discussed with catch on slack.

Adding a related issue that could benefit from this.

Left some comments on MR. mostly small stuff.

Addressed some points but not all. Left some comments on MR.

Wan't to make sure I can still review and mark so didn't go down the rabbit hole.

resolved last 2 review comments. signature made consistent with new Cron class signature.

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.