Make google_analytics_counter_cron() faster

Created on 5 December 2019, almost 5 years ago

Updated 20 June 2023, over 1 year ago

Problem/Motivation

We have a website with more than 50000 nodes. After installing the module, the cron runs take extremely long. I noticed that the google_analytics_counter_cron() function alone runs for like 10 minutes. That's because it builds every time a huge "count" queue with all the published nodes on the website. The more content we'll have, the longer it would run. Is that neccessary?

Proposed resolution

Wouldn't it be faster if this function only produced 1 "count" queue element at each cron run? This 1 queue element could do the following:

First, it queries all the published node IDs in ascending order.
It always checks if there's still time to process the next node by checking the "Queue Time (in seconds)" config setting.
If there's still time to do another one, it would run the \Drupal::service('google_analytics_counter.app_manager')->gacUpdateStorage($data['nid'], $data['bundle'], $data['vid']); just like now.
After it's done with that node, it would check if there's still time to do another one etc.
If there's no time to do another one, then it saves the last processed node ID to the state. And then ends the work in that queue element.
During next cron run, the new "count" queue element would continue the work from the following to the saved node ID.
If the queue element processed all node IDs then it would remove the saved node ID from state. And that way during the next cron run, it would start to process all nodes again.

What's your opinion about that? Could it work?

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

✨ Feature request

Status

Fixed

Version

3.0

Component

Code

Created by

🇸🇰Slovakia kaszarobert

Live updates comments and jobs are added and updated live.

Incomplete comments

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Comment over 1 year ago →
System Message

kaszarobert → authored c7d060ab on 4.0.x
Issue #3099047 by kaszarobert: Make google_analytics_counter_cron()...
Status changed to Fixed over 1 year ago12:25pm 20 June 2023
Comment over 1 year ago →
🇸🇰Slovakia kaszarobert
I decided to alter the current implementation similar to my proposal from years ago. As timeouts mentioned in referenced issue 🐛 When we have a huge number of nodes the cron job does not work (504 gateway timeout) Fixed still happening. But instead of using state, we only create 1 queue item at a time until there're no more nodes to process. And we're doing this NID descending order.
Comment over 1 year ago →
System Message
Automatically closed - issue fixed for 2 weeks with no activity.

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024