Contrib.social

Queue Processing Order for Items with Identical Source ID Values

Open on Drupal.org →
Created on 10 May 2023, almost 2 years ago

Problem/Motivation

I saw several queues that had let's say 2x items. When I ran them, they reported creating x items when I ran their migration, and then the queue reported as empty. The numbers aren't important, but it seems to be that I queued a given item type multiple times, so there could be 3x items in the queue, and only x unique items.

This got me thinking "If two items are in the queue with the same ID, do they both run? If only one runs, which value is kept?"

Steps to reproduce

I have 2 items in the queue with the same value for the source ID field value as defined by the migration.

In the queue table, the item_id column values were 145944799 and 145944800.

I manually tweaked the data column values to set the incoming data to have a title value of A for 145944799. I also set the title to B for item 145944800.

When I run the migration that uses this queue as a source, it successfully creates 1 node as expected, or updates 1 node as expected if I run it a second time after repopulating the queue with the same data.

However, the resulting node has a title of A, when I would expect/want the title to be B given the FIFO nature of a queue.

I have two theories:

  • It seems that it is running the migration from the data in item 145944799 and ignoring item 145944800
  • It is running item 145944800 before 145944799.

I'm exploring the code to try and answer this myself, but figured I would put a ticket in to gather my thoughts and see if this rang a bell for anyone else who might have encountered this previously and had advice.

Despite this little confusion, I'm loving this module. It's been a huge help with my current project.

Proposed resolution

Not sure. Perhaps we could add a setting to reverse this?

Remaining tasks

Not sure.

User interface changes

N/A

API changes

?

Data model changes

?

💬 Support request
Status

Closed: works as designed

Version

1.0

Component

Code

Created by

🇺🇸United States adamzimmermann

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @adamzimmermann
  • Comment almost 2 years ago →
    🇺🇸United States adamzimmermann
  • Comment almost 2 years ago →
    🇺🇸United States adamzimmermann

    I went through the module with xdebug and confirmed that the core queue code looked at each queue item in the order I would expect.

    However, the Drupal\migrate_source_queue\EventSubscriber\QueueItemDeleteSubscriber::onPostRowSave was only called once, which indicates to me that this is related to how the source plugin iterator defines a "row" for the migration.

    I'm beginning to wonder if this is this module working as intended or if I need to prevent "duplicates" from being put into the queue.

    However, given the nature of queues, it seems like there should way to have this module use the most recent version of an item in a queue.

    I'm going to continue to test some things in \Drupal\migrate_source_queue\QueueIterator.

  • Status changed to Closed: works as designed almost 2 years ago
  • Comment almost 2 years ago →
    🇺🇸United States adamzimmermann

    I believe I just solved this with the track_changes source property.

    https://www.drupal.org/docs/8/api/migrate-api/migrate-source-plugins/ove... → 

      source:
    track_changes: true

    I think we can close this, but hopefully this helps someone else who encounters this.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024