Contrib.social

[META] Add more tracking / analytics

Open on Drupal.org →
Created on 13 December 2023, 11 months ago
Updated 14 December 2023, 11 months ago

Problem/Motivation

For marketing purposes it can be useful to understand how subscribers are interacting with newsletters. This can include:

  1. List of email delivery failures (automatically disabling subscribers that repeatedly bounce)
  2. Percentage of subscriber who read the emails
  3. Percentage of subscribers who click links

Proposed resolution

1) We already track the total number of emails that were delivered or failed. We can add information about which subscribers fail, and disable these subscriptions so we don't repeatedly send emails that bounce. However the options are limited by the quality of the error information that this module receives: the API doesn't report the exact error codes (SMTP is OK, sendmail is terrible!), and the module doesn't handle bounce emails.

2) Read tracking likely requires tracking pixels. Increasing numbers of modern mail clients block these, and users may perceive it as spying. Perhaps it's best to avoid this??

3A) Link tracking can be done by adding a query parameter to links that indicates the newsletter issue entity ID (for links back to the site). Then we track statistics about how many times the link was followed. This is 100% privacy respecting, fast and requires minimal storage. The disadvantage is that if one subscriber clicks multiple times then each one will be counted.

3B) We can extend A) adding a second query parameter that is unique to the subscriber, generated using a one-way hash function so that it can't be traced back to a specific subscriber. We can then track also the unique IDs, and increment the link statistic only for a new unique ID. This is slower, and requires a lot more storage. It is still pretty good for privacy because the data is anonymous, but the links will resemble tracking links and it could alarm people.

Given that this module is free (libre?) software, we should be sensitive to respect the privacy rights of subscribers. I propose that

  • 1) and 3A) are definitely OK and fine to go in the main module.
  • 3B) we can leave out for now. It's debatable, it could go here or in a separate project, also it might not really be necessary as 3A) might be good enough.
  • 2) we should avoid

Remaining tasks

User interface changes

API changes

Data model changes

1) #3053111: Issue status should not be stored on issue entity →

2) Add a counter for the number of unsubscribes generate by the newsletter issue.

3) New table to track link clicking:

  • INDEX issue entity ID
  • INDEX link URL
  • count of clicks

4) In theory this module allow the newsletter entity to be of any type, but in practice it only really works with nodes. The new table is currently described without an entity type and it would be specific to nodes. This allows SQL metadata for the issue entity ID to be set correctly as a key into the node table as needed for a Drupal View.

I propose that we should make the same change to simplenews_mail_spool, and remove entity_type. If some module is sending other entity types then they would need to create spool and link tracking tables for each entity type. This matches what the general strategy for Drupal - AFAIK there tables relate to a specific entity type, e.g. node_revision etc.

🌱 Plan
Status

Active

Version

4.0

Component

Code

Created by

🇬🇧United Kingdom adamps

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @adamps
  • Comment 11 months ago →
    🇨🇭Switzerland berdir Switzerland

    I think tracking/managing bounces and similar delivery problems is useful but not trivial and often requires to actually process and parse mail responses, we worked on https://www.drupal.org/project/inmail → at some point but didn't follow-up on that.

    link and open tracking IMHO should not be a feature that this module provides. There are no privacy-respecting versions of that :) I'm well aware that all clients want that, but I think it sets a bad precedent to have that supported out of the box. If you have a client that wants that, maybe it could be a separate project?

  • Comment 11 months ago →
    🇬🇧United Kingdom adamps
  • Comment 11 months ago →
    🇬🇧United Kingdom adamps

    useful but not trivial

    Agree, and I already updated the IS before reading your comment. It's especially bad when using sendmail because sending always "succeeds". It works a little better with SMTP because more of the errors are reported immediately as send failures.

    link and open tracking IMHO should not be a feature that this module provides.

    Personally speaking I tend to agree😃. However I thought it was fair to raise the issue so that other people have a chance to comment too. I'm happy for it to be a separate project.

    Although I don't really like tracking, realistically speaking site owners want it. Either they can move to commercial platforms which often have lots of heavy tracking features, or someone else might write the add-on for simplenews. So if simplenews has a semi-official add-on for lightweight tracking that we write carefully/sensitively then maybe it's a sensible balance?? For example maybe we could try to anonymise/aggregate the information? I'd welcome your suggestions or further discussion if you have time available.

  • Comment 11 months ago →
    🇬🇧United Kingdom adamps

    Here's an idea. Instead of adding the actual subscriber ID, we could add a unique ID derived from it by a one-way hash. This would allow tracking of percentages clicked but arguably respecting privacy.

    Rather than ducking analytics entirely, I feel attracted to the challenge creating an ethical implementation that could be a show-case for others to copy.

  • Comment 11 months ago →
    🇦🇹Austria ChrisZZ Vienna

    Dear Adam,

    for us, only the percentage is important - we do not want to follow individual users. So yes - we are open to the sensitive approach.

    Best regards,
    Christian

  • Comment 11 months ago →
    🇬🇧United Kingdom adamps

    Thanks for the comments. I've updated the IS to describe some options that are 100% privacy respecting.

    @Berdir OK with you?

    I believe the best solution requires some non-BC data model changes, so it would need to wait for v5. We should allow a few months for v4 to become stable first.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024