Rebuild failed on large views

Created on 27 January 2023, almost 2 years ago

Updated 31 January 2023, almost 2 years ago

Problem

Rebuilding OAI fails on large views (thousands of items). The result is the error message:

An AJAX HTTP error occurred.
HTTP Result Code: 500
Debugging information follows.
Path: /batch?id=524&op=do_nojs&op=do
StatusText: Internal Server Error
ResponseText:

The "ResponseText:" is empty.

The rebuild fails after ten of seconds, but it depends on the settings of max_allowed_packet (MySQL) and
memory_limit (PHP). The higher the values are the later the error comes.

Trying to index 40.000+ items (max_allowed_packet=256M, memory_limit=8G) results in a very slow index
process (11 % after 12+ hours).

Having $config['system.logging']['error_level']='verbose' helps a bit, sometimes the error is>

Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away: INSERT INTO "watchdog" ...

and this error is very very long with a lot of "watchdogs".

This happens on at least two independent Drupal instances, both 9.5.2 version.

Steps to reproduce

Create an entity reference view with a large number of items to display.
Set this view to be used for OAI, sets are allowed.
Rebuild OAI manually (/admin/config/services/rest/oai-pmh/queue).

🐛 Bug report

Status

Active

Version

2.0

Component

Miscellaneous

Created by

🇨🇿Czech Republic krejcir Brno

Live updates comments and jobs are added and updated live.

rebuild

Comments & Activities

Issue created by @krejcir
Comment almost 2 years ago →
🇨🇿Czech Republic krejcir Brno
UPDATE: Setting up " Items to display: Display a specified number of items | x items" in a view, where x ~ small thousands (e. g. 5000 for 40.000+ items) speed up the OAI rebuild significantly. However, it still require quite a lot resources to finish (max_allowed_packet=256M, memory_limit=8G).
Comment almost 2 years ago →
🇨🇿Czech Republic krejcir Brno
UPDATE2: PHP's memory_limit=8G is important only when rebuild is run via GUI. Using CLI via drush queue:run rest_oai_pmh_views_cache_cron works fine even if memory_limit is much lower. max_allowed_packet still needs to be at least 256M for 30.000+ items.

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024