Problem/Motivation
Currently, when a deletion queue item is processed, it is deleted, whether the to-be-deleted content could be found in any migration maps or not.
$deletionQueueWorker->processItem($item->data);
$deletionQueue->deleteItem($item);
So what? The intent was to delete the item, and if its not in the target database (which we are assuming due to it not existing in the migrate maps), then hasn't the deletion queue item essentially accomplished its task even if it failed?
Yes and no. Here is a scenario.
1. Editor creates a new item in Orange Logic.
2. Cron queues the item for migration, but there are a lot of items to migrate, so the item is still queued to be migrated for quite a while.
3. In the meantime, the editor regrets their decision and decides to delete the item.
4. Now the item is queued for deletion and is also queued for migration. But the deletion queue, when it runs, runs top to bottom and will usually complete much sooner than an item will be migrated (if there are enough items in the queue). Essentially we have a race condition.
5. The deletion queue item tries to delete its target content, but can't find it and it gets thrown away.
6. Then the item is migrated into the database, now as an orphan, since it is in the target database but not the source database. It's source item has already been deleted, so this item will never be queued for deletion.
Steps to reproduce
(see above)
Proposed resolution
QueueWorker, by default, will remove items have `processItem()`is called. The interface does not allow for any return value. So there is no way to determine success during processing from the calling code.
However, QueueWorker does allow you to throw an Exception. If you are working with standard QueueWorker interface, QueueWorker will catch the exception, log it, and move on, but without removing the item from queue.
In addition, since in addition to the standard QueueWorker implementation, we essentially mimic this implementation in `OrangeDamQueueDataManager::processDeletedQueueItem()` we'll have to try{}catch{} there too and make sure our explicit deletion does not get called. (This is just because we have to cover both incantations:from the standard queue command interface and from our own custom commands.
Remaining tasks
Decide how to handle "stuck" orange_dam_deletion_queue items. Should there be a configurable expiration or do we force users to rely on `queue:delete orange_dam_deletion_queue` command whenever they want to clear out any stuck items?