Add a cleanup function

Created on 23 May 2025, 5 months ago

Problem/Motivation

The module currently puts no limit on the number of log entries in the system. This may lead to a very large logging table. I have a project where the table for this module is a little over 6GB.

Steps to reproduce

* Enable the module
* Generate a lot of entity operations (sorry, not quite sure how one would do this)

Proposed resolution

Implement a configuration option and a hook_cron implementation to purge log entries older than a certain age. Offer the following options:

  • Never purge (default for existing installations, in order to not introduce new behaviour)
  • 1 year
  • 180 days
  • 90 days (default for new installations)
  • 30 days
  • 2 weeks
  • 1 week
  • 1 day

Remaining tasks

  • Create MR
  • Review
  • Merge

User interface changes

There will be a new configuration option at /admin/config/system/entity_logger

API changes

None.

Data model changes

None.

Feature request
Status

Active

Version

1.0

Component

Code

Created by

🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @eelkeblok
  • Merge request !5Resolve #3526254 "Add cleanup function" → (Merged) created by eelkeblok
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    Added a quick implementation using Cursor. I am not convinced the "just delete everything older than x" through the entity interface is a great idea. Especially if there is a large backlog I don't think it will finish in a reasonable amount of time. But maybe I'm mistaken, needs a little more testing (like with a prod database from my example project :) ).

  • Pipeline finished with Success
    3 months ago
    Total: 144s
    #533010
  • Pipeline finished with Success
    3 months ago
    Total: 142s
    #535069
  • Pipeline finished with Success
    3 months ago
    #535106
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    As expected, this didn't go flawlessly, I ran into out of memory exceptions.

    • I added a batching operation, deleting 100 log entries at a time, with an cache clear for the entity storage in between.
    • I added a time limit, so the hook cron will not run longer than 10 seconds.

    Typically, I would say this is enough for most setups. Maybe it is worth mentioning somewhere in the documentation that if you consistently get warnings that the time limit was exceeded, you need to either increase the frequency the cron hook is ran (maybe with a reference to ultimate cron) or reduce the amount of log entries created.

  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱
  • Pipeline finished with Success
    3 months ago
    Total: 136s
    #535123
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    I just found 📌 Support OOP hooks Active and noticed there that currently most hooks are implemented using the OO class_resolver pattern, which makes the official OO hooks conversion much easier. So we should at least change that, possibly do it all in one go if 📌 Support OOP hooks Active lands first.

  • 🇧🇪Belgium svendecabooter Gent

    @eelkeblok: I have merged issue 📌 Support OOP hooks Active , so this functionality could be implemented with OOP hooks in mind.
    Haven't reviewed the code yet, but described functionality looks promising :)

  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    🙌 Excellent. I'll see if I can take a crack at that soon.

  • Pipeline finished with Success
    3 months ago
    Total: 182s
    #553206
  • Pipeline finished with Canceled
    3 months ago
    Total: 105s
    #553305
  • Pipeline finished with Canceled
    3 months ago
    Total: 116s
    #553306
  • Pipeline finished with Success
    3 months ago
    Total: 143s
    #553308
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    Refactored to use the new OO hook pattern.

  • Pipeline finished with Success
    3 months ago
    Total: 143s
    #553336
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱
  • 🇧🇪Belgium svendecabooter Gent

    Thanks for the MR. Logic looks good to me.
    Only thing I'm wondering: couldn't we use a QueueWorker plugin to process the deletion of the stale log entries?
    Then we wouldn't need the batch processing logic within the cron hook.
    The processing time could be configured in the QueueWorker plugin - e.g. \Drupal\locale\Plugin\QueueWorker\LocaleTranslation has this set to 30 seconds.

  • Status changed to Needs work about 2 months ago
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    I guess. We could create queue items that contain ID ranges to remove. Limiting the time spent on processing the queue is a built-in feature, so we wouldn't need to do that ourselves. We would loose the ability to report on the time limit being exceeded, which would be an indication that a system is producing more logging than can be removed. And we'd need to start to do some bookkeeping to prevent duplicate queue items being created, assuming the queue (and thereby the logging) hasn't been fully processed before the next time cron runs (which is a save assumption; why do all this, otherwise 🙂).

    Or maybe I am missing something obvious and you had something else in mind 🙂

  • 🇧🇪Belgium svendecabooter Gent

    OK those seem like 2 valid points to stick with the current approach.
    If the current logic would pose unexpected problems, we can still change the implementation later on.

  • 🇧🇪Belgium svendecabooter Gent

    Merging now.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024