Incompatibilities with message queue logic means some digested messages may never get sent

It appears that there is an incompatibility with message_digest and the "queue" feature of message_subscribe (which moves all the flag checking and notifier processing into the queue). I realize that message_subscribe is not a hard dependency for message_digest, but I believe it's a very common upstream tool used in conjunction with it. I also realize that the queue feature is not very useful when your notifier is message_digest (since it too offloads the actual sending into cron-triggerd tasks), but people still may have the message_subscribe queue option active for various reasons.

I'm jumping right into the issue queue "template" format for this one:

Problem/Motivation

When the message_subscribe queue is used, mids take an additional cron cycle (after message creation) to make their way into the message_digest table, and thus be available for sending. If this extra cycle happens to be the one right before a digest interval, the messages will fall into a "black hole" and never get sent via message_digest.

Consider a case where cron runs every 3 hours (12:00, 3:00, 6:00, 9:00, etc.) and a daily interval is used:

1. Between 9:00 and 12:00 on Monday lets say 6 messages are created which have subscribers. These messages go into the queue for subscription processing.
2. At 12:00 cron runs. For this example let's say this is also the moment in which the daily digest interval has elapsed. This means that unsent items in the message_digest table will be aggregated and sent. This happens before the queue is processed, so the 6 messages just created will not be processed, though they will make their way into the message_digest table (presumably to be taken care after the next digest interval).
3. 24+ hour elapse
4. We get to the 3:00 cron run on Tuesday (which is the next time the daily interval will be tested as elapsed). Here again unsent items in the message_digest table will be aggregated and sent. However, because MessageDigest::aggregate() filters out messages with "sent" set to FALSE and messages with a timestamp > now -1 day, our 6 messages created between 9:00 and 12:00 on Monday still won't get processed. In fact, these messages will never get processed or sent from this point forward.

Proposed resolution

There could be a few ways to deal with this:

• Explicitly claim an incompatibility between the message_subscribe queue and message_digest. As I noted this queue feature does not add much value when messages are digested anyway. It could be useful to have this formally documented.
• Modify the message_digest module weight (or hook_cron() ordering with hook_module_implements_alter()) to ensure digest aggregation runs after the queue is processed. This however will not account for cases where multiple cron runs are needed to process a queue item.
• Alter the way the "timestamp" is set in the message_digest table such that it is the time an item is added to the table instead of the time the message was created.

I'm personally a fan of that last idea.

Remaining tasks

Decide which solution makes the most sense.

User interface changes

n/a

API changes

Probably none.