Support DB transport & worker command

Looking at how Symfony achieves asynchronous message handling:

To consume messages using a separate PHP process, making it truely asynchronous, you need to run a CLI command on your production server:
php /path/to/your/app/bin/console messenger:consume async

This command is kept alive, restarted and quit by a linux service suchs as supervisord.

To achieve a similar thing in Drupal I think we should use drush. Drupal Console seems to be used much less.
So a similiar setup would be to have supervisord run
php /path/to/your/app/vendor/bin/drush messenger:consume async

Then we should translate Symfony Console commands to drush commands. Especially messenger/Command/ConsumeMessagesCommand.php

I made a first draft for async handling.

1) copied \Symfony\Component\Messenger\Command\ConsumeMessagesCommand to \Drupal\symfony_messenger\Command\ConsumeDrupalMessagesCommand
Modified the command to be Drupal compatible. Most checks and options need to be re-enabled.

2) copied core/scripts/drupal to symfony_messenger/scripts/drupal-cli
Modified the script to load the ConsumeDrupalMessagesCommand. Added the kernel boot function from \Drupal\Core\Command\ServerCommand::boot.

3) modified \Drupal\symfony_messenger\Transport\DrupalTransport and other files to make it all work:
- if you now run the controller at /symfony-messenger-example an ExampleRequest message is send to the DrupalTransport bus.
- The DrupalTransport bus saves the message as a Drupal Queue item.
- you can start the worker similar to how the Symfony Worker is started (see https://symfony.com/doc/current/messenger.html#consuming-messages-running-the-worker), for Drupal this would be > ./web/modules/contrib/symfony_messenger/scripts/drupal-cli consume-messages
- If the worker is started or the worker is running queue items are picked up and handled by ExampleRequestHandler.

Stuff to figure out:

• Use multiple busses
• Use multiple queues
• Disable cron queue handling of the messenger queue
• Make sure the worker is limited by time/number of items/memory/etc
• Make sure the worker is restarted: or leave this to supervisord (see SF messenger docs)
• Add the \Symfony\Component\Messenger\RoutableMessageBus as a service so we can use that
• Add various Symfony classes as services so we can use them: receiverLocator, busLocator, rateLimiterLocator

Ah, really nice, just what i thought was missing in the current code.

Note to self: In a followup we can steal from ultimate_cron how it initates supervisord-like process handling, or even better, leverage that module as worker.

While typing this, i wonder if we should split up the issue (transport, worker). I tend to yes.

Thanks for the project, I've been working on something similar for a while along with further integrations with Drupal. Along with a Symfony project which is powered heavily by Symfony messenger.

> As Drupal core already contains Symfony Console and Console Applications, drush is not needed.

A plain console works, one that can be run indefinitely. We should certainly avoid Drush etc, because they likely have unnecessary overhead.

The standard standard `messenger:consume` command has a good balance of features. Lets try to expose this command as best as possible. https://symfony.com/doc/current/messenger.html#consuming-messages-runnin...

I've found this command works well, even just as a php-cli image entrypoint (no need for Supervisord

---

Some feedback on MR

We should avoid trying to use the Drupal queue system as the storage for our transport. When taking a look at Doctrine-backed transports for Symfony apps, you'll notice that there are a number of columns, particularly available_at, which we cannot efficiently query with Drupal-queue database tables. Also Drupal queues may not be backed by database.

We can have our own database tables managing this.

We should provide an alternative backed transport, e.g Redis, so we can prove we are not coupled to SQL databases.

Small note on naming: `DrupalTransport` doesn't seem quite right. I'd prefer the project name in place of 'Drupal'. The internals of this would be replaced, of course.

----

The Symfony messenger pipeline is quite neat, but I don't think we should be utilizing Drupal queue as storage.

Something I've been working on, and something I'd like to propose here:

Intercept Drupal queue items as they are inserted, and passing them through Symfony messenger. Thusly eliminating the need to run Drupal-queues.

-----

On #4

> Use multiple busses

I'd say this project should have a 'default' bus, but not much more. Other buses can be added by others.

Not an issue for now, as far as I can tell.

> Use multiple queues

Per above, I'd say we stop abusing queue for transport. Instead relying on an intercept strategy.

> Use transport retry strategies

So is this referring to configuring `max_retries` in combination with `RecoverableMessageHandlingException` ?

Maybe the default bus should have a configurable values for `retry_strategy`, including the failure transport.

We should certainly support all the features for custom buses, though they can do it with code.

> Disable cron queue handling of the messenger queue

Not required if we use an intercept strategy.

> Make sure the worker is limited by time/number of items/memory/etc

If we re-use the consume command, it already has support for max items/memory.

> Make sure the worker is restarted: or leave this to supervisord (see SF messenger docs)

Leave execution strategy up to individual sites, documentation is good!

> Add the \Symfony\Component\Messenger\RoutableMessageBus as a service so we can use that
> Add various Symfony classes as services so we can use them: receiverLocator, busLocator, rateLimiterLocator, retryStrategyLocator

I don't have opinions on these [yet].

----

Perhaps this issue can be broken up for database transport and consume command.

By the way there is an `asyc` branch pushed to the main repo you may want to delete, in favor of the active MR here. Unfortunately this branch was the one I reviewed instead of the MR here (enough days had passed since checking out the project vs this branch, so it got lost in the mix)

Do you mind deleting this @lussoluca https://git.drupalcode.org/project/symfony_messenger/-/tree/asyc

I am to take a look at the extra commits here, plus push my work to this MR and elsewhere.

Hi everyone, I've pushed a new MR to https://git.drupalcode.org/project/symfony_messenger/-/merge_requests/2 . It's designed to replace MR1 here.

There is a large amount of text on the MR documenting things.

Feel free to ask q's in the MR or in Drupal Slack @ #symfony-messenger