Contrib.social

Fatal memory errors when running drush xmlsitemap:regenerate

Open on Drupal.org →
Created on 13 September 2018, about 6 years ago
Updated 2 February 2023, almost 2 years ago

Problem/Motivation

With a lot of entries in the xmlsitemap table, we have ~450,000, running drush xmlsitemap:regenerate hits memory errors on the latest 8.x-1.x-dev. I'm using Drush 9, enabled by this patch: https://www.drupal.org/project/xmlsitemap/issues/2968172#comment-12770930 → .

Here is the error and stack I hit, again this is not happening on alpha2, just the latest 8.x-1.x-dev

mmap() failed: [12] Cannot allocate memory
PHP Fatal error:  Out of memory (allocated 1041145856) (tried to allocate 20480 bytes) in /var/www/vendor/drush/drush/includes/preflight.inc on line 54
PHP Stack trace:
PHP   1. {main}() /var/www/vendor/drush/drush/drush:0
PHP   2. require() /var/www/vendor/drush/drush/drush:4
PHP   3. Drush\Runtime\Runtime->run($argv = *uninitialized*) /var/www/vendor/drush/drush/drush.php:66
PHP   4. Drush\Runtime\Runtime->doRun($argv = *uninitialized*) /var/www/vendor/drush/drush/src/Runtime/Runtime.php:41
PHP   5. Drush\Application->run($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/drush/drush/src/Runtime/Runtime.php:112
PHP   6. Drush\Application->doRun($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:148
PHP   7. Drush\Application->doRunCommand($command = *uninitialized*, $input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:248
PHP   8. Consolidation\AnnotatedCommand\AnnotatedCommand->run($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Application.php:964
PHP   9. Consolidation\AnnotatedCommand\AnnotatedCommand->execute($input = *uninitialized*, $output = *uninitialized*) /var/www/vendor/symfony/console/Command/Command.php:251
PHP  10. Consolidation\AnnotatedCommand\CommandProcessor->process($output = *uninitialized*, $names = *uninitialized*, $commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/AnnotatedCommand.php:404
PHP  11. Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter($names = *uninitialized*, $commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:150
PHP  12. Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback($commandCallback = *uninitialized*, $commandData = *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:181
PHP  13. call_user_func_array:{/var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235}(*uninitialized*, *uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235
PHP  14. Drupal\xmlsitemap\Commands\XmlSitemapCommands->regenerate(*uninitialized*) /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php:235
PHP  15. xmlsitemap_run_unprogressive_batch(*uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/Commands/XmlSitemapCommands.php:77
PHP  16. batch_process($redirect = *uninitialized*, $url = *uninitialized*, $redirect_callback = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/xmlsitemap.module:1481
PHP  17. _batch_process() /var/www/web/core/includes/form.inc:880
PHP  18. xmlsitemap_regenerate_batch_generate($smid = *uninitialized*, $context = *uninitialized*) /var/www/web/core/includes/batch.inc:294
PHP  19. Drupal\xmlsitemap\XmlSitemapGenerator->regenerateBatchGenerate($smid = *uninitialized*, $context = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/xmlsitemap.module:2473
PHP  20. Drupal\xmlsitemap\XmlSitemapGenerator->generatePage($sitemap = *uninitialized*, $page = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:320
PHP  21. Drupal\xmlsitemap\XmlSitemapWriter->generateXML() /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:192
PHP  22. Drupal\xmlsitemap\XmlSitemapGenerator->generateChunk($sitemap = *uninitialized*, $writer = *uninitialized*, $chunk = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapWriter.php:160
PHP  23. Drupal\Core\Url::fromUri($uri = *uninitialized*, $options = *uninitialized*) /var/www/web/modules/contrib/xmlsitemap/src/XmlSitemapGenerator.php:247
PHP  24. Drupal\Core\Url::fromInternalUri($uri_parts = *uninitialized*, $options = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Url.php:307
PHP  25. Drupal\Core\Path\PathValidator->getUrlIfValidWithoutAccessCheck($path = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Url.php:415
PHP  26. Drupal\Core\Path\PathValidator->getUrl($path = *uninitialized*, $access_check = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:89
PHP  27. Drupal\Core\Path\PathValidator->getPathAttributes($path = *uninitialized*, $request = *uninitialized*, $access_check = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:122
PHP  28. Drupal\Core\Routing\Router->match($pathinfo = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Path/PathValidator.php:163
PHP  29. Drupal\Core\Routing\Router->matchRequest($request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:116
PHP  30. Drupal\Core\Routing\Router->applyRouteEnhancers($defaults = *uninitialized*, $request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:130
PHP  31. Drupal\Core\Routing\Enhancer\ParamConversionEnhancer->enhance($defaults = *uninitialized*, $request = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Router.php:259
PHP  32. Drupal\Core\ParamConverter\ParamConverterManager->convert($defaults = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Routing/Enhancer/ParamConversionEnhancer.php:45
PHP  33. Drupal\Core\ParamConverter\EntityConverter->convert($value = *uninitialized*, $definition = *uninitialized*, $name = *uninitialized*, $defaults = *uninitialized*) /var/www/web/core/lib/Drupal/Core/ParamConverter/ParamConverterManager.php:100
PHP  34. Drupal\node\NodeStorage->load($id = *uninitialized*) /var/www/web/core/lib/Drupal/Core/ParamConverter/EntityConverter.php:103
PHP  35. Drupal\node\NodeStorage->loadMultiple($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php:249
PHP  36. Drupal\node\NodeStorage->doLoadMultiple($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php:279
PHP  37. Drupal\node\NodeStorage->getFromPersistentCache($ids = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php:356
PHP  38. Drupal\Core\Cache\DatabaseBackend->getMultiple($cids = *uninitialized*, $allow_invalid = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php:941
PHP  39. Drupal\Core\Cache\DatabaseBackend->prepareItem($cache = *uninitialized*, $allow_invalid = *uninitialized*) /var/www/web/core/lib/Drupal/Core/Cache/DatabaseBackend.php:122
PHP  40. unserialize(*uninitialized*) /var/www/web/core/lib/Drupal/Core/Cache/DatabaseBackend.php:167
PHP  41. drush_shutdown() /var/www/vendor/drush/drush/includes/preflight.inc:0
PHP  42. error_get_last() /var/www/vendor/drush/drush/includes/preflight.inc:54

Proposed resolution

Investigate what the cause of the memory issues are. Perhaps introduce queuing or something to chunk the work?

Remaining tasks

  • Identify the memory issues
  • Implement a work around and/or bring down memory requirements

User interface changes

None

API changes

Mayyyybe

Data model changes

Mayyyybe

🐛 Bug report
Status

Fixed

Version

1.0

Component

xmlsitemap.module

Created by

🇨🇦Canada m4olivei Grimsby, ON

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment almost 2 years ago →
    🇺🇦Ukraine ankondrat4 Lutsk

    Hello.

    I have a big project with more 150000 nodes and when I try run rebuild XML Sitemap and regenerates the cached files from UI, batch doesn't finish success.

    Errors from logs:

    [02-Feb-2023 07:32:33 Europe/Paris] PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /mnt/www/html/docroot/core/lib/Drupal/Core/Entity/Query/Sql/Tables.php on line 263 request_id="v-fc6c2b6c-a2c2-11ed-b923-9b540a5d00ce"

[02-Feb-2023 08:20:02 Europe/Paris] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 1048576 bytes) in /mnt/www/html//docroot/core/lib/Drupal/Core/Entity/EntityStorageBase.php on line 221 request_id="v-a09a9254-a2c9-11ed-89a5-cbc98f92183c"

[02-Feb-2023 11:37:15 Europe/Paris] PHP Fatal error:  Maximum execution time of 120 seconds exceeded in /mnt/www/html//docroot/core/lib/Drupal/Core/Field/BaseFieldDefinition.php on line 606 request_id="v-fe405f0e-a2e4-11ed-beae-0faeab71a96d"

[02-Feb-2023 12:05:00 Europe/Paris] PHP Fatal error:  Maximum execution time of 150 seconds exceeded in /mnt/www/html//docroot/core/lib/Drupal/Core/Database/Query/Select.php on line 812 request_id="v-a094f848-a2e8-11ed-9135-ef7aaf4987cf"

Feb  2 11:23:06 [2244]: Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /mnt/www/html/docroot/core/lib/Drupal/Core/Database/Query/Condition.php on line 405
Feb  2 11:23:06 [2244]: Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 40960 bytes) in /mnt/www/html//vendor/composer/ClassLoader.php on line 571 in Consolidation\SiteProcess\ProcessBase->getOutputAsJson() (line 171 of /mnt/www/html//vendor/consolidation/site-process/src/ProcessBase.php).

    I tried from console run drush xmlsitemap:regenerate but doesn't finish success too.
    Errors:

    In ProcessBase.php line 171:
                                                                                                                    
  Unable to decode output into JSON: Syntax error                                                                   
                                                                                                                    
  Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /mnt/www/html/  
  /docroot/core/lib/Drupal/Core/Database/Query/Condition.php on line 319                                    
                                                                                                                    
  Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 65536 bytes) in /mnt/www/html/  
  /vendor/symfony/http-kernel/Event/TerminateEvent.php on line 24

    Patch #15 has realized in last version of module, but it didn't resolve issues with memory as I see. Maybe someone has new ideas how fix this?

  • Comment almost 2 years ago →
    🇺🇦Ukraine ankondrat4 Lutsk
  • Comment almost 2 years ago →
    🇺🇦Ukraine ankondrat4 Lutsk
  • Comment about 1 month ago →
    🇺🇸United States selwynpolit

    I found that by setting the `Number of links in each sitemap page` to 5000 or less, my memory problems went away. This is accessible via the `/admin/config/search/xmlsitemap/settings` URL and is hidden under the `Advanced settings`

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024