Inefficient notification-callback can block all available processes

Created on 4 January 2023, over 2 years ago
Updated 15 November 2024, 5 months ago

Problem/Motivation

When performing a bulk delete or change in Brightcove to hundreds of videos (via API or otherwise) causes a swam of notification-callbacks to the module from Brightcove.

These requests are serialised, and only one will run at any moment in time, and all other requests will block. With a php-fpm configuration or threaded configuration with 200 processes or threads, it would only take 500 video deletes or updates in quick succession to cause the pool to fill up and all processes be blocked by semaphore waits.

Steps to reproduce

1. Use API with parallel threading to delete 1000 videos
2. Site will go offline with the swarm of notification callbacks

Proposed resolution

Enqueue the notification callbacks to a queue, and process them in cron, and completely remove semaphore waits

Remaining tasks

-

User interface changes

-

API changes

-

Data model changes

-

Feature request
Status

Needs work

Version

3.1

Component

Code

Created by

🇬🇧United Kingdom Driskell

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇭🇺Hungary yce

    Seems like a feature request. It works as designed.

    The original goal was to prevent overwhelming Drupal by introducing a semaphore to hold off multiple simultaneous requests from Brightcove as even uploading a single video could cause desynchronization/inconsistency without a semaphore.
    If I remember correctly the notification subscription support was introduced later in the module, and the mass create/delete was not considered as an issue at the time.

    Although moving from away from semaphore to using queue to process the notifications could slow down the sync even more as every queue item needs to be processed in order to achieve full synchronization.

    The queue system is already overwhelmed with the entity sync.

Production build 0.71.5 2024