Add Symfony Messenger Integration Module

Created on 11 December 2024, 7 months ago

Problem/Motivation

Create a new module (ai_messenger) that provides Symfony Messenger integration for background processing of AI operations, with Redis transport support and queue monitoring capabilities.

Long-running AI operations currently block the user interface and can timeout during execution. We need:

  • Asynchronous processing of AI operations
  • Reliable message queuing
  • Progress tracking for background tasks
  • Queue monitoring and management
  • Ability to handle process interruptions

Proposed resolution

  1. New Module Structure
    • Create ai_messenger module
    • Implement Redis transport configuration
    • Add queue monitoring capabilities
    • Provide integration with Process Abstraction Layer
  2. Core Features
    • Message queue management
    • Background worker processes
    • Progress tracking integration
    • Queue status monitoring
    • Error handling and recovery
  3. Configuration System
    • Redis connection settings
    • Worker process configuration
    • Queue settings
    • Retry policies
  4. Monitoring Features
    • Queue status overview
    • Failed message handling
    • Process monitoring
    • Worker status tracking

Remaining tasks

Explore https://www.drupal.org/project/redis β†’

API changes

  • New service definitions
  • New configuration schema
🌱 Plan
Status

Active

Version

1.0

Component

Other Submodules

Created by

πŸ‡¬πŸ‡§United Kingdom seogow

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @seogow
  • πŸ‡¬πŸ‡§United Kingdom MrDaleSmith

    No problem with this idea, but it may make more sense as its own project rather than being part of the core AI module.

  • πŸ‡¬πŸ‡§United Kingdom scott_euser

    Haven't looked at this in any detail, but to note there is https://www.drupal.org/project/sm β†’ . I believe the idea is that the Queue system in Drupal can be leveraged and what is then used to processing queues is handled e.g. by that. In case there is any duplication of efforts

  • Issue was unassigned.
  • πŸ‡ΊπŸ‡ΈUnited States Kristen Pol Santa Cruz, CA, USA

    We are doing some issue management housekeeping and adding/removing components.

    We are removing the "Other Submodules" component and want people to categorize issues with the best module/submodule component.

    Moving this issue to "...to be triaged" for a closer look by the team.

    See πŸ“Œ Update AI module project components Active for more details.

  • πŸ‡ΊπŸ‡ΈUnited States Kristen Pol Santa Cruz, CA, USA
  • πŸ‡©πŸ‡ͺGermany marcus_johansson

    I have discussed this with Luca that has worked on the SM module and JΓΌrgen and its obvious that it will help solve some long running task problems we currently have.

    I will set this to 2.0.0 and postpone, because it will require some architectural changes.

Production build 0.71.5 2024