I'm trying to figure out how to throttle a queue so that only a certain number of items are processed each time Ultimate Cron runs.
Using hook_cron(), I fetch 28,000 records, and then I separate the records into manageable chunks by using $queue->createItem(). The worker callback then syncs those records with existing entities on my site.
define('MY_API_BATCH_SIZE', 100);
/**
* Implements hook_cron()
*/
function my_api_cron() {
// Only if the queue is empty should we re-fetch new items.
$queue = DrupalQueue::get('my_api_sync_records', TRUE);
$queue_length = $queue->numberOfItems();
if ($queue_length == 0) {
// Only allow this to run every 24 hours.
$last_sync = variable_get('my_api_last_sync', 0);
if ($last_sync > strtotime('-24 hours', REQUEST_TIME)) {
return;
}
variable_set('my_api_last_sync', REQUEST_TIME);
// Populate queue
$my_api = new MyAPI();
$records = $my_api->fetch_all();
watchdog('My API', 'Fetched @count records.', array('@count' => count($records)), WATCHDOG_INFO);
// Break the record-set into manageable chunks.
for ($i = 0; $i < count($records); $i += MY_API_BATCH_SIZE) {
$item = array('records' => array_slice($records, $i, MY_API_BATCH_SIZE), 'changed' => REQUEST_TIME);
$queue->createItem($item);
}
}
}
/**
* Implements hook_cron_queue_info().
*/
function my_api_cron_queue_info() {
return array(
'my_api_sync_records' => array(
'worker callback' => '_my_api_cron_worker',
'time' => 30,
),
);
}
/**
* Worker callback for my_api_cron_queue_info().
*
* Sync all records.
*/
function _my_api_cron_worker($item) {
// Sync all $item['records'] to entities in our system.
foreach ($item['records'] as $record) {
// ... Do sync stuff: sync records to entities, and mark when
// each entity was last changed using $item['changed'].
}
}
Ultimate Cron is configured to run the queue processor every minute, and according to my logs, it processes between 5-10 queue items at a time (i.e. 500-1,000 records). Is there a way to throttle/force only a certain number of queue items to be processed at a time?
On second thought, maybe I'm misunderstanding the whole point of "queue items", and I should not be breaking the record set into multiple queue items, and instead just have a single queue item with all records, and the 'time' key in my_api_cron_queue_info() should be sufficient to let a certain number of records process at a time. Is this the correct approach?
Any thoughts are appreciated!
Closed: outdated
2.0
Documentation
Not all content is available!
It's likely this issue predates Contrib.social: some issue and comment data are missing.
I'm closing this since Drupal 7 is now unsupported and there will be no more development on the Drupal 7 branch of Ultimate Cron either.
Thank you for taking the time and effort in reporting this issue, even though it never got resolved.