Large number of log entries leads to memory/timeout issues

Created on 25 April 2023, about 1 year ago
Updated 9 May 2023, about 1 year ago

Problem/Motivation

When there are thousands of log entities the cron run to delete old logs runs into out of memory errors and timeouts issues.

Steps to reproduce

When there are thousands, perhaps 10,000 plus, log entries running the cron to delete the old logs errors.

Proposed resolution

Refactor deleteExpiredLoggedEntities() to load and delete one log entity at a time instead of loading all of the entities at one time.

Add a new field to set a limit on the number of log entities to delete per cron run.

Remaining tasks

User interface changes

API changes

Data model changes

πŸ› Bug report
Status

Fixed

Version

2.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States kamkejj WI

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @kamkejj
  • Open on Drupal.org β†’
    Core: 9.5.5 + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    Waiting for branch to pass
  • @kamkejj opened merge request.
  • Status changed to Needs work about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States apmsooner

    @kamkejj, a few comments...

    • The new field type should maybe be a number field so can be validated through html5 widget? This means min, max, step properties.
    • If it's timing out with large number, 1000 probably is still way too high a default number. Maybe something like 50 would be adequate? An entity load of anything more than that seems like it could still be a heavy load.
    • Need to add the new field to config/schema/gatsby.schema.yml and config/install/gatsby.settings.yml
    • Need to add new field to tests/src/Functional/AdminFormTest
  • πŸ‡ΊπŸ‡ΈUnited States kamkejj WI

    Deleting 1000 takes ~20 seconds on Pantheon's dev environment. I don't think that's too long, but if you think the default should be lower I can change it.

    I'll about getting the other changes completed.

  • Open on Drupal.org β†’
    Core: 9.5.5 + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    Waiting for branch to pass
  • Status changed to Needs review about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States kamkejj WI

    Andy, made the updates and set the default to 500 from 1000.

  • Open on Drupal.org β†’
    Core: 9.5.5 + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    Waiting for branch to pass
  • Open on Drupal.org β†’
    Core: 9.5.5 + Environment: PHP 7.4 & MySQL 5.7
    last update about 1 year ago
    Waiting for branch to pass
  • Status changed to Fixed about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States apmsooner

    Merged to dev.

  • Status changed to Fixed about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States apmsooner
Production build 0.69.0 2024