Overwhelmed Pull Queue is permanent because Cron times out

Created on 1 February 2018, almost 7 years ago
Updated 19 July 2023, over 1 year ago

We have a site with an extremely large number of records connected to an existing mapping. That mapping was configure for push-only until recently.

We updated the configuration to use pull recently, and now have an untenable pull-queue issue. The "Last pull date" is, of course, set to some arbitrarily old date, and every Salesforce record in the system has been updated since then.

In the Salesforce_pull QueueHandler: "getUpdatedRecordsForMapping()", "enqueueAllResults" is called before "setLastPullTime()". However, none of these functions are sensitive to the number of results coming back from Salesforce, and whether that total might be untenable. When this function times out, the "setLastPullTime()" doesn't run. That means it doesn't get changed, ever. That means that as soon as the queue gets small enough to add items, we are back where we started, pulling in every record in the system.

I'm not sure the best way to handle this. Perhaps the SF api allows us to limit the total number of results, and we can sort them by date, then use the most recent date as a new value? That seems ugly.

Anyway, to work around this problem I need a way to change that updated date in the mapping history. We already have a drush operation for reseting it, so I adapted that to add a function simply setting it arbitrarily.

πŸ› Bug report
Status

Fixed

Version

3.0

Component

salesforce_pull.module

Created by

πŸ‡ΊπŸ‡ΈUnited States gcb

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Production build 0.71.5 2024