Make dblog entities

Created on 4 January 2015, almost 11 years ago
Updated 27 January 2023, over 2 years ago

Problem/Motivation

Dblog module hasn't really changed since Drupal 6. The upgrade to Drupal 8 left the dblog Controller with a lot of SQL queries inside. People looking for inspiration in Drupal core, should not use dblog as an example of how to organize the code of a Drupal 9+ project.

Also, there is no way to access dblog entries without directly querying the database, and dblog is usually found in tests where modules want to ensure a given log is present after a certain action. There is no Dblog Entry API either, so everything dblog related is coupled to the table structure.

Proposed resolution

There are a lot of parts of drupal core that directly query against the watchdog table. Also contributed modules do this as well. In order to reduce the friction of this upgrade the proposal is:

  • Keep using the Dblog Logger service to insert logs, so no performance is lost when creating logs. Also make the DblogEntry class final, so no contrib module can extend it. This will reduce the need of potential breaking changes.
  • Convert Dblog Entries into content entities. But make these entities read only in the sense they cannot be updated after being created. This is what someone would expects from immutable log entries.
  • Be as conservative as possible by Initially making this entity internal. Because Drupal core assumes all entities can be saved. This will, however, make jsonapi integration not available out of the box.
  • Provide an EntityListBuilder to reduce the code of the DblogController. Will also simplify the views integration. and potentially expose the entity via jsonapi or rest modules. Provide a ViewBuilder to render the log details.
  • When uninstalling dblog, dblog entities will be automatically deleted.

Regarding BC compatibility:

  • Leave DblogController untouched. Only change references in the the dblog.routing.yml to load the entity collection. By providing a ListBuilder and a ViewBuilder we should be able to replicate the same functionality as currently provided by the dblog module. The current set of automated tests should pass for the new implementation using ListBuilder and ViewBuilder.
  • Direct queries to the watchdog table are still valid, but not recommended anymore

Remaining tasks

Reviewers needed! All code is done and test are in place.

  • Add the new DblogEntry class [✅ Done]
  • Add the new DblogEntryStorage class [✅Done]
  • Add the View Builder [✅ Done]
  • Add the List Entity Builder. [✅ Done]
  • Add the remaining methods the DblogEntryStorageInterface. [✅ Done]
  • Add new tests (without deleting the existing ones). [✅ Done]
  • Add Upgrade path for watchdog view [✅ Done]
  • Get some in depth code reviews. [❌ pending]
  • Discuss the topics mentioned below [❌ pending]

To Discuss

The following items are some architectural decisions that require some consensus from the community.

1. Are chosen names for the DblogStorageInterface methods are good enough?

  • DblogEntryStorageInterface::deleteAll(int $keep = 0);
  • DblogEntryStorageInterface::loadMostRecent(string $channel = '');
  • DblogEntryStorageInterface::messageTypes();
  • DblogEntryStorageInterface::mostFrequentLogEntries(array $properties);

2. Are we ok with forbidding individual deletion of entries?

3. Are we ok with preventing saving an DblogEntry using save() and suggests instead use the Dblog logger?

User interface changes

None. Some internal classes will change, instead of having dblog-error to render the icons, we will rely on numeric values provided by RfcLogLevel. For example severity-4.

API changes

  • New DblogEntry class.
  • New DblogEntryInterface interface.
  • New DblogEntryStorage class.
  • New DblogEntryStorageInteface interface.
  • New DblogEntryAccessControllerHandler class.
  • New DblogViewsData class.
  • New DblogEntryListBuilder class.
  • New FormatMessage service.

Data model changes

None. The watchdog table will remain intact.

Original report

By @chx: We could make dblog entities with a separate service writing the base data tables for fast writing.

📌 Task
Status

Needs review

Version

10.1

Component
Database Logging 

Last updated 4 months ago

Created by

🇨🇦Canada chx

Live updates comments and jobs are added and updated live.
  • Needs framework manager review

    It is used to alert the framework manager core committer(s) that an issue significantly impacts (or has the potential to impact) multiple subsystems or represents a significant change or addition in architecture or public APIs, and their signoff is needed (see the governance policy draft for more information). If an issue significantly impacts only one subsystem, use Needs subsystem maintainer review instead, and make sure the issue component is set to the correct subsystem.

  • needs profiling

    It may affect performance, and thus requires in-depth technical reviews and profiling.

Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Merge request !2747Make dblog content entities → (Open) created by dagmar
  • 🇺🇸United States smustgrave

    Still very interested in this.

    Wonder if you could provide a good way for one to test this?

  • Status changed to Needs work over 2 years ago
  • 🇺🇦Ukraine voleger Ukraine, Rivne

    Added review comments

  • First commit to issue fork.
  • Status changed to Needs review over 2 years ago
  • 🇨🇳China jungle Chongqing, China
  • 🇩🇪Germany mmbk Meißen

    The failing test with postgres is not generated by this patch, it already fails with the 10.1.x-dex branch https://www.drupal.org/node/3060/qa

  • Status changed to Needs work over 2 years ago
  • 🇨🇳China jungle Chongqing, China

    NW for #75

    1. Are chosen names for the DblogStorageInterface methods are good enough?

    No opinion.

    2. Are we ok with forbidding individual deletion of entries?

    Any reason for this? -1.

    3. Are we ok with preventing saving an DblogEntry using save() and suggests instead use the Dblog logger?

    Make sense.

  • 🇮🇳India gauravvvv Delhi, India
  • Status changed to Needs review over 2 years ago
  • 🇨🇳China jungle Chongqing, China

    Per the test result of DbLogEntryTest::testSerialization() #75 may not be an issue anymore. Probably, it's not an issue in newer versions of PostgreSQL.

  • Status changed to Needs work over 2 years ago
  • The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

    Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

    Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

  • 🇬🇧United Kingdom catch

    Left some comments on the MR. More than profiling I am wondering if the indexes are correct (i.e. an EXPLAIN on some common queries would be useful) and whether the schema matches between existing sites and a new install.

  • Assigned to dagmar
  • 🇦🇷Argentina dagmar Argentina

    I'm back to work on this.

  • Status changed to Needs review over 2 years ago
  • 🇦🇷Argentina dagmar Argentina

    I updated the issue summary to answer @catch concerns regarding indexes. I ran a `DESCRIBE watchdog;` command before and after installing the site using this patch, and the table structure remains the same.

    I guess this also is valid for the issue 📌 Make dblog entities Postponed , if we fix this in the dblog.install module, the changes will also apply to dblog as entities.

    I also addressed the comments about dblog exception in ContentUninstallValidator. Based on my current understanding of the module_installer logic, there is no way for dblog to by pass this content validator exception, so I had to modify it any way. I tried to make it as simple and clear possible, and not only for dblog.

  • 🇦🇷Argentina dagmar Argentina
  • Status changed to Needs work over 2 years ago
  • The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

    This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

    Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

  • Status changed to Needs review over 2 years ago
  • 🇦🇷Argentina dagmar Argentina
  • 🇦🇷Argentina dagmar Argentina

    I updated the issue summary based on the latest reviews assuming the proposed method names are ok. Removing needs profiling as I did the profiling tests in #61.

  • Issue was unassigned.
  • 🇦🇷Argentina dagmar Argentina

    I went over all the pending open threads of the merge request and closed most of them. I left a few open to make sure the provided answer is enough for the original reviewers.

    I also checked some concerns I had related caching. It seems entities are not cached but routes are (example route:[language]=en:/admin/reports/dblog/event/10) in cache_data table. This is also happening for the current non entity implementation so I will not attempt to fix this as part of this issue. Maybe is not even an issue.

    Most of the concerns regarding performance, indexing and future compatibility with 🐛 Database primary keys can exceed maximum integer storeable (has actually occurred, in watchdog) Fixed are addressed as the underlying system stills uses the same database scheme and insert queries.

    My general feeling about the code (as the current dblog maintainer) is: this is pretty much ready to be in. There is some room of improvement in the DblogEntryViewBuilder, but probably qualifies for a follow up. And comparing this with the current code of the dblog module I would say this is a big improvement in code quality and alignment of Drupal standards. And will also allow to close several postponed issues for more than a few years now.

    The remaining steps in my opinion are:

    1. Reviewing the latest changes to ContentUninstallValidator::validate
    2. as this is calling a new API method: ContentEntityTypeInterface;:requireDeleteContentBeforeUninstall which needs some grammar and common sense checking.

    3. Review if all annotations defined in DblogEntry are correct, or if something is missing or redundant.
    4. Get some last code review round paying specially attention to the high level picture of this. I already did this a few times, but I wrote most of the code, and doesn't feel right to assume it is right.
  • Status changed to Postponed over 2 years ago
  • 🇦🇷Argentina dagmar Argentina

    I had more time to think about item 1. I think we should not make an exception for dblog entries. After researching a bit why the content validator was created it seems a bit dangerous to bypass the checks.

    Concerns about disabling the dblog module in a production environment are still valid though. I created this to take care of this functionality: Provide a way to disable creation of database log entries Needs work

  • Hmm, idk why it doesn't show my picture or link to my profile. That's weird.

  • Merge request !8723Make dblog entities → (Open) created by dagmar
  • Pipeline finished with Failed
    over 1 year ago
    Total: 189s
    #220044
  • Pipeline finished with Failed
    over 1 year ago
    Total: 144s
    #220056
  • Pipeline finished with Failed
    over 1 year ago
    Total: 206s
    #220059
  • Pipeline finished with Failed
    over 1 year ago
    Total: 167s
    #220067
  • Pipeline finished with Failed
    over 1 year ago
    Total: 715s
    #220068
  • Status changed to Needs review over 1 year ago
  • 🇦🇷Argentina dagmar Argentina

    I found a way to not have to pause the creation of logs and remove the dblog module transparently as usual.

    The approach is not elegant, but it works. When dblog module is asked if has data from the ContentUninstallValidator it replies no... And the module can be uninstalled. Logs are deleted anyways by the dblog_module_preuninstall hook. There is no other way to implement that I'm aware of as the ModuleInstaller::uninstall method does not offer any other opportunity to modify the validations.

    This means this issue is not longer postponed on Provide a way to disable creation of database log entries Needs work

  • Pipeline finished with Success
    over 1 year ago
    Total: 743s
    #220106
  • 🇦🇷Argentina dagmar Argentina

    dagmar changed the visibility of the branch 2401463-dblog-entities-D10 to hidden.

  • 🇦🇷Argentina dagmar Argentina

    dagmar changed the visibility of the branch 2401463-dblog-entities to hidden.

  • 🇦🇷Argentina dagmar Argentina

    Added Allow kernel tests to fail or expect logged errors Needs work to the issue summary to, to proper fix the problem with tests. Thanks @joachim

  • Status changed to Needs work over 1 year ago
  • The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

    This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

    Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

  • Pipeline finished with Success
    about 1 year ago
    Total: 2832s
    #280269
  • Status changed to Needs review about 1 year ago
  • 🇦🇷Argentina dagmar Argentina
  • Status changed to Needs work about 1 year ago
  • 🇺🇸United States smustgrave

    Added some comments to the MR. Majority small stuff.

Production build 0.71.5 2024