Add support for using queues for sending submissions to CiviCRM

Created on 15 July 2025, about 1 month ago

Problem/Motivation

Let's add support for using queues for sending submissions to CiviCRM. This is useful to speed up form submissions and to make sure failed submissions aren't lost. Note that when using queues, you can't use [return-data] tokens or redirects set by CiviCRM.

Proposed resolution

Add a new 'Use queue for sending submissions to CiviCRM' setting with 3 options: 'Always', 'Never' and 'Only when initial submission failed'. The default value is 'Never'.

Feature request
Status

Active

Version

2.2

Component

Code

Created by

🇧🇪Belgium dieterholvoet Brussels

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @dieterholvoet
  • Merge request !42Add support for using queues → (Open) created by dieterholvoet
  • 🇳🇱Netherlands jaapjansma

    I am not sure whether I understand this issue correctly and that is because the CiviCRM McRestFace (CMRF) Framwork already provides queuing. To queue a call for later execution one can set to be scheduled.

    I also believe that the CiviCRM McRestFace has the ability to retry failed calls however I am not sure whether that is fully functional.

    To explain a bit more about the CiviCRM McRestFace Framework. It is a platform agnostic approach for calling the CiviCRM Api. Implementations exists for at least wordpress, Drupal 7 and Drupal 8/9/10/11.

    The platform agnostic part is called CMRF Abstract Core and can be found here: https://github.com/CiviMRF/CMRF_Abstract_Core

    The Drupal 8/9/10/11 implementation (inlcuding configuration screens) for the Abstract Core is called cmrf_core and can be found here: https://www.drupal.org/project/cmrf_core

  • 🇧🇪Belgium dieterholvoet Brussels

    As discussed in the call:

    • calls are stored in the civicrm_api_call table
    • calls can be scheduled to be executed at a later time
    • calls can be retried N times after failing

    This abstract functionality is provided by the cmrf_abstract_core package. Let's try to keep using Drupal queues, but use the civicrm_api_call table as storage backend. If that turns out to be too difficult we can still consider dropping Drupal queues completely, or to use the current implementation that uses the Drupal queue storage.

    An advantage of using Drupal's queue backend is that you can e.g. use Redis instead of a database table. We're not currently using that though, so not super important.

    An advantage of using CMRF's queue backend is that it integrates with the /admin/reports/cmrfcalls overview, which is a useful interface for the client.

    I'll have another look at this MR soon.

Production build 0.71.5 2024