Better support for Derivative queue workers

Created on 10 May 2024, 7 months ago

Problem/Motivation

I have an application with more than 300 (and counting) derivative queues, making Queue UI difficult to use.

Addition, Queue UI incorrectly reports "zero" for all derivative queues.

Steps to reproduce

Create a derivative queue.

Proposed resolution

Group derivative queues in the UI under a base queue worker umbrella, and allow to be inspected and processed as a single queue.

Remaining tasks

User interface changes

API changes

Data model changes

πŸ“Œ Task
Status

Active

Version

3.1

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States AaronBauman Philadelphia

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @AaronBauman
  • πŸ‡ΊπŸ‡ΈUnited States AaronBauman Philadelphia

    Not sure what I was looking at, but the item count for subqueues looks fine in fact.

  • Status changed to Postponed: needs info 7 months ago
  • πŸ‡ΊπŸ‡¦Ukraine voleger Ukraine, Rivne

    Can you provide an example of a derivative queue definition for a test case?

  • πŸ‡ΊπŸ‡ΈUnited States AaronBauman Philadelphia

    Here's the deriver i'm using for the curious:

    /**
     * @file
     * Contains \Drupal\media_analysis\Plugin\Derivative.
     */
    
    namespace Drupal\media_analysis\Plugin\Derivative;
    
    use Drupal\Component\Plugin\Derivative\DeriverBase;
    use Drupal\Core\Entity\EntityStorageInterface;
    use Drupal\Core\Plugin\Discovery\ContainerDeriverInterface;
    use Drupal\media_analysis\HookImplementations;
    use Symfony\Component\DependencyInjection\ContainerInterface;
    
    /**
     * Provides block plugin definitions for nodes.
     *
     * @see \Drupal\media_analysis\Plugin\QueueWorker\MediaAnalysisWorker
     */
    class MediaAnalysisWorker extends DeriverBase implements ContainerDeriverInterface {
    
      /**
       * Constructor.
       *
       * @param \Drupal\Core\Entity\EntityStorageInterface $node_storage
       *   The node storage.
       */
      public function __construct(private readonly EntityStorageInterface $nodeStorage) {
      }
    
      /**
       * {@inheritdoc}
       */
      public static function create(ContainerInterface $container, $base_plugin_id) {
        return new static(
          $container->get('entity_type.manager')->getStorage('node')
        );
      }
    
      /**
       * {@inheritdoc}
       */
      public function getDerivativeDefinitions($base_plugin_definition) {
        $nodes = $this->nodeStorage->loadByProperties(['type' => 'media_channel']);
        foreach ($nodes as $node) {
          $this->derivatives[$node->id()] = $base_plugin_definition;
          $this->derivatives[$node->id()]['admin_label'] = t('Channel: ') . $node->label();
        }
        return $this->derivatives;
      }
    
    }
    

    This deriver then gets referenced in the @Cron annotation field of my existing cron queue worker:

    /**
     * Defines 'media_analysis' queue worker.
     *
     * @QueueWorker(
     *   id = "media_analysis",
     *   title = @Translation("Media Analysis: Channels"),
     *   cron = {"time" = 60},
     *   deriver = "\Drupal\media_analysis\Plugin\Derivative\MediaAnalysisWorker"
     * )
     */
    

    And I have another procedure to insert queue items with a queue name like media_analysis:$nid which then get processed by the queue worker, thanks to Derivative API.

    The derivative is mostly useful so that I can more easily prevent duplicate queue items, but it breaks the Queue UI pretty badly.

    I started looking at adapting Queue UI, but it's a bit of a can of worms - will require special handling for every route and callback that uses the {queueName} placeholder, for starters.
    So IMO it's fine to leave this issue on Postponed, and I'll implement a custom UI for my special case.

  • Assigned to voleger
  • Status changed to Needs work 7 months ago
  • πŸ‡ΊπŸ‡¦Ukraine voleger Ukraine, Rivne

    So, you expect to see a separate view based on the selected base derivative worker definition, right? I see that as a separate route with the same table rener approach for the overview page, but queues are created based on the selected derivative.

    I'll prepare the MR with the approach I see so you can try it out.

  • πŸ‡ΊπŸ‡¦Ukraine voleger Ukraine, Rivne

    @aaronbauman, please try patching the 3.2.x version of the queue_ui module with the diff of MR. Let me know whether it is a usable iteration for now.

    I'm okay with the current implementation and will add the functional test coverage. However, I need feedback on the current version of the feature.

  • Pipeline finished with Skipped
    2 months ago
    #302721
    • voleger β†’ committed 74971479 on 3.2.x
      Issue #3446581 by voleger, aaronbauman: Better support for Derivative...
  • πŸ‡ΊπŸ‡¦Ukraine voleger Ukraine, Rivne

    I added test coverage for grouping. It has to be stable. In case of any issues, create a follow-up.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024