Segmentation fault during cron

I think this could be the at fault code, where its working with a large array in memory.
https://git.drupalcode.org/project/google_analytics_counter/-/blob/4.0.x...

I see on the screenshot that it downloaded quite a few paths someday. Did it work on the same production environment up until now? Does this crash happen constantly or just randomly? Indeed, it seems a memory related issue. But the cause of that is what's gonna be hard to tell. Here are a few ideas I'd check now:

Some limit is not enough in php.ini?
Out of physical memory?
One of the installed PHP extensions?
Or the A weird APCu bug?
Could the google/analytics-data library which this module uses cause some trouble with certain server configurations?

The debug cron logging pasted here shows that it used at some point 110.85 MB RAM which I think not that big number considering how many things the cron job is downloading and writing to db. I'd look into phpinfo on the webserver and try to compare the php ini settings with the other (probably dev) environment where you stated that this problem does not occurs if there are some major differences in some memory or cache limits.

An update, our host PHP memory limit is 128mb.

@kaszarabert Thanks for the ideas and I will use them to investigate more, It stopped working across all our Acquia environments, and they actually had different code deployed, so I think it could be an issue with the amount of data being retrieved from Google is my current theory. Since nothing on the servers changed at the point in time when things broke, at least that I have found yet.

Will research more and update tomorrow.

Still debugging, added a bunch of logging to the current cron logic, looks like the problem lies in the queryTotalPaths()

[info] Starting execution of google_analytics_counter_cron(), execution of file_cron() took 4.12ms. [19.61 sec, 64.75 MB]
 [notice] Starting cron with memory: 71827456 [19.62 sec, 64.82 MB]
 [notice] Before node count query [19.62 sec, 64.83 MB]
 [notice] Total nodes: 16481. Memory: 71827456 [19.63 sec, 64.92 MB]
 [notice] Starting main processing. Memory: 71827456 [19.63 sec, 64.92 MB]
 [notice] Before database truncate. Memory: 71827456 [19.64 sec, 64.92 MB]
 [notice] After database truncate. Memory: 71827456 [20.23 sec, 64.99 MB]
 [notice] Before queryTotalPaths. Memory: 71827456 [20.23 sec, 64.99 MB]
Segmentation fault

The problem seems to lie with the serialize() in the buildQuery method, not really sure where I go from here to debug further....

 [info] Starting execution of google_analytics_counter_cron(), execution of file_cron() took 2.59ms. [19.88 sec, 64.77 MB]
 [notice] Starting cron with memory: 71827456 [19.89 sec, 64.86 MB]
 [notice] Before node count query [19.89 sec, 64.86 MB]
 [notice] Total nodes: 16480. Memory: 71827456 [19.9 sec, 64.95 MB]
 [notice] Starting main processing. Memory: 71827456 [19.9 sec, 64.95 MB]
 [notice] Before database truncate. Memory: 71827456 [19.91 sec, 64.95 MB]
 [notice] After database truncate. Memory: 71827456 [20.52 sec, 65.02 MB]
 [notice] Before queryTotalPaths. Memory: 71827456 [20.52 sec, 65.02 MB]
 [notice] Starting queryTotalPaths [20.52 sec, 65.02 MB]
 [notice] About to buildQuery [20.53 sec, 65.02 MB]
 [notice] Starting buildQuery [20.53 sec, 65.02 MB]
 [notice] Got config [20.53 sec, 65.02 MB]
 [notice] Chunk: 1, Pointer: 0 [20.54 sec, 65.02 MB]
 [notice] Built query dates: Array
(
    [start] => 2024-10-22
    [end] => 2024-11-20
)
 [20.54 sec, 65.02 MB]
 [notice] Property ID: 371690843 [20.54 sec, 65.02 MB]
 [notice] Starting cache options creation [20.55 sec, 65.15 MB]
 [notice] Params structure: Array
(
    [property] => properties/371XXXX43
    [dateRanges] => Array
        (
            [0] => Google\Analytics\Data\V1beta\DateRange Object

        )

    [dimensions] => Array
        (
            [0] => Google\Analytics\Data\V1beta\Dimension Object

        )

    [metrics] => Array
        (
            [0] => Google\Analytics\Data\V1beta\Metric Object

        )

    [offset] => 0
    [limit] => 1
)
 [20.55 sec, 65.15 MB]
 [notice] Before serialize [20.55 sec, 65.15 MB]
Segmentation fault

PROBLEM:

Had a segmentation fault when trying to serialize() Google Analytics API objects (DateRange, Dimension, Metric)
These objects can't be safely serialized in PHP, causing the crash

SOLUTION:

Create two separate parameter arrays:

$parameters: Contains proper Google Analytics objects needed for the API call
$cache_params: A simple array with basic data types that can be safely serialized

WHY:

We only need serialization to create a unique cache ID (md5 hash)
The actual API call requires proper Google Analytics objects
By separating these concerns, we avoid the segfault while maintaining functionality

So going to make that proposed change, Claude.ai helped debug the issue.

UGH now getting

 [info] Starting execution of google_analytics_counter_cron(), execution of file_cron() took 3.68ms. [21.11 sec, 64.77 MB]
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1732220684.750398     339 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1732220684.753480     339 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
Segmentation fault

Ok So after a lot more debugging on and off with acquia support, its something related to the Google Vendor gax, I found this comment that talks about a very hard to debug segmentation fault! And I can agree it was very hard :)

https://github.com/googleapis/gax-php/blob/140599cf5eae2432363ce6198e9fd...

And I was able to add enough debugging to pull this out of the codebase

[debug] Getting feed [37.63 sec, 116.37 MB]
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1736874324.703588 708 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1736874324.703698 708 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
[error] Error: Maximum call stack size of 10436608 bytes (zend.max_allowed_stack_size - zend.reserved_stack_size) reached. Infinite recursion? in Google\ApiCore\CredentialsWrapper->Google\ApiCore\{closure}() (line 244 of /var/www/html/vendor/google/gax/src/CredentialsWrapper.php) #0 [internal function]: Google\ApiCore\CredentialsWrapper->Google\ApiCore\{closure}()
#1 /var/www/html/vendor/grpc/grpc/src/lib/UnaryCall.php(43): Grpc\Call->startBatch()
#2 /var/www/html/vendor/grpc/grpc/src/lib/BaseStub.php(295): Grpc\UnaryCall->start()

And then I found the hosting I am using has these PHP differences

grpc
grpc support => enabled
grpc module version => 1.68.0
grpc.enable_fork_support => 0 => 0
grpc.grpc_trace => no value => no value
grpc.grpc_verbosity => no value => no value
grpc.poll_strategy => no value => no value
fiber.stack_size => no value => no value
fiber.stack_size => no value => no value
zend.max_allowed_stack_size => 0 => 0
zend.reserved_stack_size => 0 => 0

Still don't have a solution, but at least the problem is narrowed down a lot more.

Great to hear you somehow found out new info about this problem. I was just checking if this library has some issues that's related to these segmentation faults and stack errors and I found this comment at https://github.com/googleapis/gax-php/issues/584#issuecomment-2342772917:

We also encountered this problem. PHP 8.3 has new feature and tries to detect the recursion. This feature likely doesn't work correctly with the gRPC PHP extension.

The workaround is to disable stack size checks in the PHP INI settings:
zend.max_allowed_stack_size: -1.

Can you check if this is set in php.ini, httpd.conf, or .htaccess, then what happens?

The second more work heavy option would be doing the needed Analytics API call manually without Google's PHP library with CURL or Guzzle somehow. So making a new setting in the module where you can switch implementations. These errors in the library are all related to some gRPC stuff but I wonder if the auth and the pageview query can be done via a simpler REST API call.

Thanks I tried the suggested "zend.max_allowed_stack_size: -1" but did not help :(

Were still trying to fix this, but got a new breakthrough, after many attempts to force the Google/gax to use REST instead of GRPC we still were getting a seg fault (seemed to be around accessing the credentials file), BUT we found once we removed the PHP package protobuf https://github.com/protocolbuffers/protobuf our seg faults went away, so that appears to be causing the issue.

# ddev config.yaml
webimage_extra_packages: [
  "pngcrush",
  "libjpeg-progs",
  "ffmpeg",
  "clamav",
  "php${DDEV_PHP_VERSION}-dev",
  "php${DDEV_PHP_VERSION}-grpc",
  "php${DDEV_PHP_VERSION}-oauth",
  "php${DDEV_PHP_VERSION}-http",
  "php${DDEV_PHP_VERSION}-gmp",
  "php${DDEV_PHP_VERSION}-dba",
  "php${DDEV_PHP_VERSION}-gnupg",
  "php${DDEV_PHP_VERSION}-protobuf",  <--- This is the problem, once removed and "ddev restart" and seg fault is fixed.
  "php${DDEV_PHP_VERSION}-pspell",
  "php${DDEV_PHP_VERSION}-raphf",
  "php${DDEV_PHP_VERSION}-rdkafka",
  "php${DDEV_PHP_VERSION}-ssh2",
  "php${DDEV_PHP_VERSION}-yaml"
]

Wow, you did an amazing progress. So does the module's current codebase work with the Google libraries when no "php${DDEV_PHP_VERSION}-protobuf" is installed? Or do we need to commit some changes you did previously in the MR for fixing this problem completely?

Also, I think we should create an issue with this for the protobuf maintainers, so they hear about this, too.

nicholass → closed merge request !21

Full Disclosure I would not have gotten this fixed without Calude.ai but see the new MR22 which has in it a fix that fixes the segfault issue. I will paste an AI explanation from Claude below, as this is a bit over my head.

Steps to reproduce.

1. Add protobuf https://github.com/protocolbuffers/protobuf to your environment, I used ddev as noted above

webimage_extra_packages: [
  "php${DDEV_PHP_VERSION}-dev",
  "php${DDEV_PHP_VERSION}-grpc",
  "php${DDEV_PHP_VERSION}-protobuf"
]

2. Run cron see seg fault.

Claude.ai Explanation as follows:

The root cause involved three interacting issues:

Protobuf Objects and Serialization

When we tried to serialize Google Analytics' response data, we were dealing with special Protobuf objects. These objects aren't regular PHP objects - they're optimized for network communication using Google's Protocol Buffer format. Trying to serialize them directly caused conflicts between PHP's native serialization and Protobuf's internal serialization mechanisms, leading to segmentation faults.

Memory Management

The original approach tried to handle these Protobuf objects as regular PHP arrays, which created memory management problems. The segmentation faults occurred because we were trying to access memory in ways that the Protobuf extension wasn't expecting, especially when trying to serialize these objects for caching.

RepeatedField Type Safety

The final error revealed that we were treating a Google\Protobuf\Internal\RepeatedField (a special Protobuf container type) as a PHP array. This type mismatch caused PHP's array functions to fail because they expect native PHP arrays.

After testing on our Acquia hosting, this does appear to fix our issue, and I now see GRPC logging after a cron run with debug.

drush cron -vvv --debug

 [info] Starting execution of google_analytics_counter_cron(), execution of webform_scheduled_email_cron() took 5.41ms. [47.08 sec, 110.3 MB]
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1737655590.172511     303 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1737655590.176159     303 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
 [info] Execution of google_analytics_counter_cron() took 650.16ms. [47.73 sec, 111.84 MB]
I0000 00:00:1737655651.551433     245 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1737655651.553439     245 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
 [info] Retrieved 7350 items from Google Analytics data for paths 1 - 7350. [109.36 sec, 150.36 MB]
 [info] Merged 7349 paths from Google Analytics into the database. [138.03 sec, 150.71 MB]
 [info] Cron run completed. [138.05 sec, 150.4 MB]

I can't thank you enough for your commitment and professionality for finding the root cause and making a fix to this problem. Who knew the protobuf extension would cause this much problem with objects.

The changes look fine, so I committed them. Now there are some weird things happening with tests which are unrelated but I will fix them in another issue.

Thank you for your help again!

Anytime, glad to have such a responsive maintainer!

Automatically closed - issue fixed for 2 weeks with no activity.