Implement a mass-email-sending protection (frequency capping)

Created on 1 July 2024, 2 months ago
Updated 7 August 2024, about 1 month ago

Problem/Motivation

If a mass of log entries appear in Drupal, it may happen that watchdog_mailer created far too many emails, that for example cause outgoing email blocking at server level. Essentially, that may mean, that important other regular emails from Drupal (like registration) can not be sent.

For that reason, we should discuss and implement a mechanism to limit the outgoing emails at a high enough limit (frequency capping).
I also thought about bundling similar mails, but that doesn't really make sense to me for log entries, because they at least differ in time and user typically and in most cases you may want to have the information immediately!

So I'd suggest implementing some kind of counter that caps the amount of emails sent to a maximum per time and as last email sends an information about the capping executed, so the recipients can take a look into the logs manually.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Feature request
Status

Fixed

Version

3.0

Component

Code

Created by

🇩🇪Germany Anybody Porta Westfalica

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @Anybody
  • 🇩🇪Germany Anybody Porta Westfalica
  • 🇩🇪Germany LRWebks Porta Westfalica

    Okay, let's talk about some concrete questions here:

    If we view frequency capping as [Limit] per [Timeframe] then:

    1. What should the user be setting in the Form? Amount, Time, or both? The outgoing mails server limit probably differs a lot depending on what server is employed, correct? So I suppose setting either just the amount or both is feasible.
    2. How should we keep track of the amount of mails sent in the last [Timeframe]? Do we do it via a database table, like the previous "Max emails per hour" setting handled it? Perhaps there is a different way, like a service that is able to count mails sent during the last [Timeframe]? That would also take into account any mails sent from different modules or core, since e.g. if the server caps at 100 mails/hour, yet a different module has already sent 100 mails, we can't send 100 mails too.
    3. What should we do after [Timeframe] has expired? For one, we need to reset the counter to 0, but do we send the mails that the admin has missed previously due to the limit or do we start fresh and only mail newly occurring logs?
  • 🇩🇪Germany Anybody Porta Westfalica

    Re #3:

    • We have to put our logic in front of sending these emails, so all the logic is inside our module, before passing them to the mail() method. To be more precise, I see two methods:
      • Frequency capping (stop mailing after sending X notifications already)
      • Delayed sending (do not send emails for a certain amount of time and then send a digest)
    • Yes [Limit] per [Timeframe] is correct. We're only looking at our (not yet sent) log mails.

    As log notifications are time-critical, I think we need to use frequency capping, so we should stop sending further notifications if we sent X mails per timeframe already. A bonus would be to not send the same email too frequently, because it might not be as helpful to see the same message a thousand times, but to see different messages instead.

  • 🇩🇪Germany LRWebks Porta Westfalica
  • Assigned to LRWebks
  • Status changed to Needs work about 2 months ago
  • Pipeline finished with Failed
    about 2 months ago
    Total: 164s
    #227820
  • Pipeline finished with Failed
    about 2 months ago
    Total: 156s
    #231877
  • Pipeline finished with Failed
    about 2 months ago
    Total: 168s
    #231918
  • Pipeline finished with Failed
    about 2 months ago
    #231933
  • Pipeline finished with Success
    about 2 months ago
    Total: 157s
    #231970
  • Pipeline finished with Success
    about 2 months ago
    Total: 157s
    #231975
  • Pipeline finished with Success
    about 2 months ago
    Total: 167s
    #232033
  • Pipeline finished with Success
    about 2 months ago
    Total: 170s
    #232124
  • Issue was unassigned.
  • Status changed to Needs review about 2 months ago
  • 🇩🇪Germany Anybody Porta Westfalica
  • Status changed to Needs work about 2 months ago
  • 🇩🇪Germany Anybody Porta Westfalica

    Nice @LRWebks I left some final comments, afterwards this LGTM and we should tag a new release after merge.

  • Pipeline finished with Success
    about 2 months ago
    Total: 171s
    #232797
  • Status changed to Needs review about 2 months ago
  • 🇩🇪Germany LRWebks Porta Westfalica

    Alright, all remaining issues resolved now. @Anybody are you taking a final look?

  • 🇩🇪Germany Anybody Porta Westfalica

    GREAT JOB @LRWebks! :) 🎉🎉🎉🚀🚀

  • Pipeline finished with Skipped
    about 2 months ago
    #233029
  • Status changed to Fixed about 2 months ago
  • 🇩🇪Germany Anybody Porta Westfalica
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024