Timeout on large sites nodes during install

Created on 25 October 2023, about 1 year ago
Updated 2 August 2024, 4 months ago

Problem/Motivation

On large sites for example with 16k content nodes I am having an error due to trying to populate the URL alias table all in one php process, I hit a message about 60 second timeout during install, or on cloud environments deployment logs such as acquia a "killed" is shown in the logs.

Steps to reproduce

Have a large site, install the module.

Proposed resolution

Use the batch API to chunk out all the work of populating the views url alias table.

- Transition to Queue API to populate views URL alias table
- D10 watchdog fix
- Entity misspelling fixed
- Remove duplicates before adding paths aliases to Queue

Remaining tasks

User interface changes

API changes

Data model changes

πŸ› Bug report
Status

Needs review

Version

3.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States NicholasS

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @NicholasS
  • πŸ‡ΊπŸ‡ΈUnited States NicholasS

    So I tried batch and it works over the UI but does not populate during a drush cim install of the module.

  • @nicholass opened merge request.
  • πŸ‡ΊπŸ‡ΈUnited States NicholasS

    So tested this locally and MR 16 is just the Queue API change, but I had another issue open and need that as well so that is MR 17 since I can't have 2 patches pathing the same lines.

    Please review MR 16

  • Status changed to Needs review about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States NicholasS
  • πŸ‡ΊπŸ‡ΈUnited States NicholasS
  • πŸ‡ΊπŸ‡ΈUnited States NicholasS

    I have done a lot of testing on my site and I think MR17 should be reviewed it fixes multiple things with this module and works as intended.

  • First commit to issue fork.
  • πŸ‡¨πŸ‡¦Canada joel_osc

    Great patch everyone! Necessary on my site in order to use this module. I noticed that some nodes could not be found by alias, in looking at it I found the queueing code was storing paths that it had already done and checking that before queueing. They key the code used did not have the langcode of the path so I was only getting each node in one of two languages. Small fix committed above.

  • πŸ‡¨πŸ‡¦Canada dstorozhuk Chicago πŸ‡ΊπŸ‡Έ, Toronto πŸ‡¨πŸ‡¦, Kyiv πŸ‡ΊπŸ‡¦
  • πŸ‡¨πŸ‡¦Canada dstorozhuk Chicago πŸ‡ΊπŸ‡Έ, Toronto πŸ‡¨πŸ‡¦, Kyiv πŸ‡ΊπŸ‡¦

    Queue options might not work for people who has cron disabled for some reason.
    I think the right option here is Batch operation for views_url_alias_rebuild_path(). But also module installation should use batch somehow.

  • πŸ‡§πŸ‡ͺBelgium michaelsoetaert

    I've rerolled the changes from MR#17 on branch 8.x-2.x-with-issue-3396154 on the latest version of the 3.x branch.

    We needed multilingual support (different URL aliases for different translations), which branch 3.x provides, but we were also getting timeouts because of the size of the website (where the changes in this issue come into play).

  • πŸ‡§πŸ‡ͺBelgium michaelsoetaert

    Sadly, the patch in comment #13 πŸ› Timeout on large sites nodes during install Needs review still resulted in timeouts on our higher environments (due to different PHP values). The issue seemed to be the large amount of data being loaded in views_url_alias_rebuild_path, since it's retrieving the complete Entity-object of each path alias.

    I decided to try the approach @dstorozhuk suggested (using the Batch API). Only loading the path alias IDs in views_url_alias_rebuild_path, splitting the list in chunks and only loading the Entity-objects of the given path alias IDs in each batch operation. That seems to have fixed the timeouts.

    Attached patch with the described functionality.

Production build 0.71.5 2024