Modify the queue item depending on the http response code

I'm working with Pankaj at Greenwich Council and we're trying to extend webform_handler_defer to have some more advanced options for responding to failures from the external api calls.

I'll create an issue fork here so we can experiment with some options.

Hi @finn-lewis and @pankajroyal

My thoughts on how this could be done - I hope this is some help:

Add a record back into the module's queue - more or less what's done by the existing defer() method in WebformDeferredProcessor.php. The other handler (the one making the API request) could write directly to the webform_handler_defer_submissions queue, but of course the right way would be some callable method inside the Webform Handler Defer module - probably added to WebformDeferredProcessor.

Potentially there could be more than one handler, per submission, that needs to be retried - ideally use a single queue record for all of them, but using multiple queue records would probably be simpler.

Don't retry until some some reasonable time has passed. Probably, store a timestamp in the queue record - earliest time to retry. Add logic to processSubmissions() to make it skip over any records which should not be retried yet. Maybe also some maximum number of retries or maximum time.

Thanks @andy-inman

Add a record back into the module's queue - more or less what's done by the existing defer() method in WebformDeferredProcessor.php. The other handler (the one making the API request) could write directly to the webform_handler_defer_submissions queue, but of course the right way would be some callable method inside the Webform Handler Defer module - probably added to WebformDeferredProcessor.

At the moment I am capturing the response code in the processor so it does not fit the single responsibility. I think the deferred handler should add the task the task back to queue in case of failure. I will refactor that.

Potentially there could be more than one handler, per submission, that needs to be retried - ideally use a single queue record for all of them, but using multiple queue records would probably be simpler.

For us, I would like this to be in single record as in my case they should happen in sync (other handlers are tied up to first handler with API calls success/fail response)

Don't retry until some some reasonable time has passed. Probably, store a timestamp in the queue record - earliest time to retry. Add logic to processSubmissions() to make it skip over any records which should not be retried yet. Maybe also some maximum number of retries or maximum time.

Exactly! I have added count and the webform_id in the queue data but next steps is to update retry count and the queue name to reflect webform id (so the tasks can be separated as batch or bucket). Alternatively I can use semaphores too.

I am struglling with the visibility of the Drupal queue, I have tried ultimate_cron and queue_ui but I cant the the list of tasks in the queue. Going to try rabitmq with Drupal queue.

Hi @pankajroyal , thanks for the contribution, I'll review it when I can.