Add a cleanup function

Created on 23 May 2025, 2 months ago

Problem/Motivation

The module currently puts no limit on the number of log entries in the system. This may lead to a very large logging table. I have a project where the table for this module is a little over 6GB.

Steps to reproduce

* Enable the module
* Generate a lot of entity operations (sorry, not quite sure how one would do this)

Proposed resolution

Implement a configuration option and a hook_cron implementation to purge log entries older than a certain age. Offer the following options:

  • Never purge (default for existing installations, in order to not introduce new behaviour)
  • 1 year
  • 180 days
  • 90 days (default for new installations)
  • 30 days
  • 2 weeks
  • 1 week
  • 1 day

Remaining tasks

  • Create MR
  • Review
  • Merge

User interface changes

There will be a new configuration option at /admin/config/system/entity_logger

API changes

None.

Data model changes

None.

Feature request
Status

Active

Version

1.0

Component

Code

Created by

🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @eelkeblok
  • Merge request !5Resolve #3526254 "Add cleanup function" → (Open) created by eelkeblok
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    Added a quick implementation using Cursor. I am not convinced the "just delete everything older than x" through the entity interface is a great idea. Especially if there is a large backlog I don't think it will finish in a reasonable amount of time. But maybe I'm mistaken, needs a little more testing (like with a prod database from my example project :) ).

  • Pipeline finished with Success
    28 days ago
    Total: 144s
    #533010
  • Pipeline finished with Success
    25 days ago
    Total: 142s
    #535069
  • Pipeline finished with Success
    25 days ago
    #535106
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    As expected, this didn't go flawlessly, I ran into out of memory exceptions.

    • I added a batching operation, deleting 100 log entries at a time, with an cache clear for the entity storage in between.
    • I added a time limit, so the hook cron will not run longer than 10 seconds.

    Typically, I would say this is enough for most setups. Maybe it is worth mentioning somewhere in the documentation that if you consistently get warnings that the time limit was exceeded, you need to either increase the frequency the cron hook is ran (maybe with a reference to ultimate cron) or reduce the amount of log entries created.

  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱
  • Pipeline finished with Success
    24 days ago
    Total: 136s
    #535123
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    I just found 📌 Support OOP hooks Active and noticed there that currently most hooks are implemented using the OO class_resolver pattern, which makes the official OO hooks conversion much easier. So we should at least change that, possibly do it all in one go if 📌 Support OOP hooks Active lands first.

  • 🇧🇪Belgium svendecabooter Gent

    @eelkeblok: I have merged issue 📌 Support OOP hooks Active , so this functionality could be implemented with OOP hooks in mind.
    Haven't reviewed the code yet, but described functionality looks promising :)

  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    🙌 Excellent. I'll see if I can take a crack at that soon.

  • Pipeline finished with Success
    3 days ago
    Total: 182s
    #553206
  • Pipeline finished with Canceled
    3 days ago
    Total: 105s
    #553305
  • Pipeline finished with Canceled
    3 days ago
    Total: 116s
    #553306
  • Pipeline finished with Success
    3 days ago
    Total: 143s
    #553308
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱

    Refactored to use the new OO hook pattern.

  • Pipeline finished with Success
    3 days ago
    Total: 143s
    #553336
  • 🇳🇱Netherlands eelkeblok Netherlands 🇳🇱
Production build 0.71.5 2024