Update to 3.5 replaces purge_drush but does not disable it

Digging in a bit this issue is occurring when numberOfItems is called in web/modules/contrib/purge/src/Plugin/Purge/Queue/DatabaseQueue.php.

When I run cron it is run twice and the first time it returns an integer as expected, the second time it returns nothing and the error is thrown.

It also happens when config import happens and there is config imported

This is killing several pipelines. After further investigation it occurs on drush commands updb, cim, or cron when something needs to execute. So anything related to queues.

It occurs when a Varnish purger is set, if I delete the varnish purger entity then it works as expected.

I think I forgot one big thing, this started after updating to Drupal 10

Same for us - similar problem.

PHP Fatal error:  Uncaught Error: Call to a member function prepare() on null in /var/www/web/core/lib/Drupal/Core/Database/StatementWrapper.php:51
Stack trace:
#0 /var/www/web/core/lib/Drupal/Core/Database/Connection.php(630): Drupal\Core\Database\StatementWrapper->__construct()
#1 /var/www/web/core/modules/mysql/src/Driver/Database/mysql/Insert.php(42): Drupal\Core\Database\Connection->prepareStatement()
#2 /var/www/web/modules/contrib/purge/src/Plugin/Purge/Queue/DatabaseQueue.php(60): Drupal\mysql\Driver\Database\mysql\Insert->execute()
#3 /var/www/web/modules/contrib/purge/src/Plugin/Purge/Queue/QueueService.php(235): Drupal\purge\Plugin\Purge\Queue\DatabaseQueue->createItem()
#4 /var/www/web/modules/contrib/purge/src/Plugin/Purge/Queue/QueueService.php(201): Drupal\purge\Plugin\Purge\Queue\QueueService->commitAdding()
#5 /var/www/web/modules/contrib/purge/src/Plugin/Purge/Queue/QueueService.php(345): Drupal\purge\Plugin\Purge\Queue\QueueService->commit()
#6 /var/www/web/modules/contrib/purge/modules/purge_drush/src/Commands/PurgeDrushCommands.php(1131): Drupal\purge\Plugin\Purge\Queue\QueueService->destruct()
#7 [internal function]: Drupal\purge_drush\Commands\PurgeDrushCommands->__destruct()

We resolved it by downgrading to previous version 3.4.0.
Site runs on the latest Drupal 9 version

I agree with @pasive. I'm on Drupal 10 and the issue happened when I upgraded the module to version 3.5. Downgrading to the previous version resolved the issue for me.

Did you downgrade to the rc or to 3.4?

Nevermind, the RC and 3.5 are identical, so it must be 3.4

I went through the changes that occurred between 3.4 and 3.5 and I have a theory. First I grabbed the queue changes, I don't think these are the cause of the issue, but I'll list them here for convenience:
https://git.drupalcode.org/project/purge/-/commit/75afeb4fc2b28c8e782410...
https://git.drupalcode.org/project/purge/-/commit/a21be456066026d96c4462...
https://git.drupalcode.org/project/purge/-/commit/3a6a46f76d20617bbb9827...

I suspect the issue is this though: https://git.drupalcode.org/project/purge/-/commit/0cdda1784393ddd55bc1aa...

purge_drush was deprecated, and part of that was hiding the module, but nothing was done in an update hook to disable it so it's still enabled on sites, it just does not show up in the list. I suspect that hiding the module and adding the functions to the core module, but not uninstalling it is what is causing the issues.
https://git.drupalcode.org/project/purge/-/commit/0cdda1784393ddd55bc1aa...

I tested this by manually disabling purge drush and I do not get the error any longer.

Huh interesting.... the reason why its not removed is that when you delete a submodule it will cause a WSOD without manual db intervention. Therefore for the entire 3.x branch, the module needs to remain, even if we have it hidden.

The reason why we didn't automatically disable it is because sites could have been relying on the old namespaces, and disabling the module will kill their code. But in theory it should have worked along side the new modules even if enabled.

For API purposes, it should be using the libraries within purge itself. So its odd that its failing. I'll prioritize some time in the next weeks to look at it.

Disabling it manually seemed to fix the issue though I had to add the drush processor again to my config.

In an effort of trying to clean up the big picture a bit let me summarize the situation:

1. Between the dates when versions 3.4 → (Sep 14, 2022) and 3.5 → (Oct 12, 2023) were released more than an entire year has passed with lots of changes (commit history). Therefore not fully clear to me why @nicxvan → highlighted only those 4 commits in #13 🐛 Update to 3.5 replaces purge_drush but does not disable it Needs work out of the 14 in total. Maybe he is right but I still suspect all changes until getting a deeper understanding of the issue.
2. Also, I feel unsure whether the issue reported in #9 🐛 Update to 3.5 replaces purge_drush but does not disable it Needs work and joined in #10 🐛 Update to 3.5 replaces purge_drush but does not disable it Needs work stems from the same root as the originally reported one. @pasive → and @alfattal → could you please test and confirm that upgrading to version 3.5, then manually disabling the purge_drush submodule (as it worked for @nicxvan in #15 🐛 Update to 3.5 replaces purge_drush but does not disable it Needs work ) eliminates the error on your environments as well? If it does then we can keep believing that your issue relates to the originally reported one.
3. Do we agree that any other contrib projects (eg. varnish_purge as mentioned above) are involved in causing the issue? Or was it only an early guess from OP which became obsolete to the current state of our knowledge?
4. Finally, in order to be able to tackle the issue (or issues? – as keeping the question open if we're facing the same bug) we definitely need to fixture the exact circumstances, and then compose a list of steps for reproducing the exception.

@all if you could paste here in a comment the result of the `$ drush pdia` command that would help a lot to see in which combination are our Purge configurations set up.

@Balu Ertl → In my case, I can confirm that disabling purge_drush and then, upgrade to 3.5 eliminated the error. I tried first to upgrade and then, manually disable purge_drush, but that DIDN'T eliminate the error.

For number 1 I highlighted those cause they had to do with the queue service and drush.
I suspect 9 and 10 are related cause they are all queue service
I don't think purge_varnish is related, I think that was a red herring

I think the steps to reproduce would be:

Install 3.4
Enable purge_drush
Set up drush processor
Enable varnish purge
Set up purger
Use core queuer and database queue
Upgrade to 3.5
Run drush cron or anything that would process the queue.

I try to fix the issue by relocating the update hook enforcing the cache clear from the sub-module to the parent if it's being invoked with a higher chance during an update process.

@all would you mind testing on your environments if this solves the exceptions?

I cannot test since I manually updated my config. I don't think this will work though. I think you need something that clears definitions like this:
https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Plugin%21...

Since the plugin location changes.
We might need to uninstall the purge_drush module instead of just hiding it too.
I'm not sure if config needs to be changed.

I think this issue is tangentially related: https://www.drupal.org/i/3391383 → -- we need to handle disappearing plugins better. Maybe some of the work in the patch there can be used to make handling the missing drush plugins uninstall correctly?

That makes sense.

Just to clarify how I fixed it on my site.
1. Remove processor options from my purgers for all env (using config split)
2. Uninstall purge_drush
3. Add the drush processor back to all env (using config split)
4. Export config and save for all env

This fixed the issue for my site.

Anywhere we're creating a new instance of the plugin, ($this->pluginManager->createInstance) we're likely needing to wrap them:

• IteratingServiceBaseTrait.php
• InvalidationsService.php
• PurgersService.php --> note the Akami patch already does this
• QueueService.php
• PluginTestBase.php

Although @japerry said in #14 → :

“The reason why we didn't automatically disable it is because sites could have been relying on the old namespaces, and disabling the module will kill their code […]”

After a couple of days of exploration, I would revisit the above statement. In light of the number of people who had expressed their difficulties due to keeping the sub-module still installed, I believe that those who still rely on a piece of non-functional placeholder code should be a smaller minority. So I concluded that playing by Drupal's own terms is the safest we can do here. Just as ticket title suggests my solution disables the then-non-existent purge_drush sub-module during the update process.

Steps as I tested:

1. Get the fresh source code:
   • for you, it would be $ composer update drupal/purge
   • for me during testing it was $ composer require "drupal/purge:dev-3397227-update-to-3.5 as 3.x-dev" -W
2. Ensure that Drupal's internal mechanisms are notified about the changes: $ drush updb OR $ drush devel:reinstall purge
3. Try the original Drush command if it works (should! – for me at least does): $ drush p:queue-work

My last commit technically automates the steps that @nicxvan has shared in #27 → with the difference that you don't need to manually disable the sub-module before the update process.

Thanks, does this add the new drush processor automatically?

As it turned out during our recent call with Jakob, I was missing one key information: the reason why he would prefer avoiding uninstalling the sub-module. If some custom or contrib code depends on these Drush commands, then that would cause them a functionality loss. Of course, we can not know if any custom code relies on them among the 30k installation base. However, regarding the contrib space: I did minimal research and went through the modules appearing on the Purge ecosystem → list. Only two projects, akamai and purge_control offer any Drush command plugins but none of them has anything to do with our sub-module.

@nicxvan Now tested again. Performing the steps above (and $ drush cc to empty the plugin cache) leaves the processor plugin instance intact. The only change in site configuration you should see is the removal of this line:

diff --git a/config/sync/core.extension.yml b/config/sync/core.extension.yml
index 7038ab9..d1646f8 100644
--- a/config/sync/core.extension.yml
+++ b/config/sync/core.extension.yml
@@ -44,7 +44,6 @@ module:
   path: 0
   path_alias: 0
   purge: 0
-  purge_drush: 0
   purge_processor_cron: 0
   purge_processor_lateruntime: 0
   purge_purger_http: 0

Just clearing cache after deploying 3.5 to the server does resolve the issue for us. We did not disable purge_drush module.

I spent some time on this today and there are a few things to point out:
1) The original 3.5 code enabled ALL drush plugins by default. Only 3 were supposed to be 'enable_by_default':

drush_purge_queue_add
drush_purge_queue_work
drush_purge_invalidate

I don't -think- enabling the other plugins by default would cause the errors in the database query, but its untested, so who knows that could be it.

2) The 'enable_by_default' is sorta messy with purge. It means plugins are enabled by the mere existence of the module. This means that for those using the above drush commands (and associated plugins), they will DISAPPEAR if you disable purge_drush. You must manually re-enable the modules via the other drush commands (p:processor-add drush_purge_queue_work) or via purge ui.

3) The purge_drush module, with the above PR (purge_drush_fix) only serves to enable those three plugins by default.

4) The plugin cache gets automatically rebuilt when you run updb and have purge_drush enabled or if you manually disable purge_drush and run manually run updb. The classes are pointing to the purge core src directory. The command classes in purge_drush effectively do nothing, with the exception of those who may have extended them in their own custom commands and added annotations.

I'll give this a few weeks (since its christmas 2023) and then commit it if I don't see any responses.

I think one thing to keep in mind is purge_drush should at least be unhidden and marked as deprecated / remove.
One of the reasons it took so long for me to debug initially was the module was hidden from the module pages cause it's marked as hidden.

I've tested the two PRs and still getting the error message when running drush cron or drush p:queue-work.

@balu-ertl → Regarding your comment in #18 🐛 Update to 3.5 replaces purge_drush but does not disable it Needs work , I think you're right. After revisiting this issue, I realized that it was caused by an old patch → that is still sitting in my composer.jason file. After removing this patch, I was able to upgrade to version 3.5 without any issue.