Enhancements to subscriber history tracking

Might not be the right issue, but starting here.

I'm supporting a client that is migrating from Simplenews 2.x to 4.x They had a custom export that tracked how many users subscribed and when they subscribed. They are trying to rewrite that to use the history table instead, and that's mostly working.

One thing we noticed that seems to be missing is any information on what the change actually was. It just tracks when and which newsletters were affected. But the information whether this was a subscribe or unsubscribe seems to be completely missing. It has a source, but that's just a route it seems (used to be a descriptive action like subscribe or bulk subscribe?)

So once someone subscribed/unsubscribed more than once, it seems impossible to reconstruct what change was actually done and query for that. Maybe you could go through all history changes of a subscriber and assume the first was subscribe but that seems unreliable.

This would need to be stored per subscription as you could subscribe and unsubscribe things at the same time. Or we could track it per history entry and then log two changes if it's a mix?

And about the mail change and similar cases, I'm not sure why history isn't directly referencing a subscriber through an ID. Having to do joins between history and a subscriber would need to go through the e-mail now, which, if that changes, would be impossible to match/show.

The original issue was 📌 Track history of subscribe/unsubscribe and proof of consent Fixed . I didn't find anyone to discuss it with at the time so I did my best😃

> used to be a descriptive action like subscribe or bulk subscribe?
actually it used almost always to be "website" 😃

> It has a source, but that's just a route it seems
Why "only" a route. A route seems like an excellent way to pinpoint the source. It's an existing concept with a unique ID and a localised human-readable name. The unique ID doesn't currently have an accessor function but you can write $this->get('source')->value;

Aha I understand now - you wish to know about the changes per newsletter subscription. The current implementation is very similar to an entity revision. True, it doesn't directly store differences, however they can be computed easily - just search backwards for the previous history entity with the same email address and compare the subscriptions field.

I wasn't attracted to the idea of a "changes" field because it's just duplication of information and it can't be generalised to other fields such as email address and in future we could make the history fieldable to match the subscriber fields.

There's also hook_simplenews_subscribe().

> I'm not sure why history isn't directly referencing a subscriber through an ID

The subscription history is independent of the Subscriber object, so it remains after that object is deleted. This could even be used to support "right to be deleted", we could use an index based on a hash of the email value to prevent re-subscription without have to store the original email address.

Possibly we could add the subscriber ID, however code would need to allow for it not being set. My guess is that developers will mostly fail to do so😃.

> I didn't find anyone to discuss it with at the time so I did my best😃

Yes yes, got the hint ;)

Thanks for the feedback.

Route as source may be describe, but it's impossible to query/process that in a useful way, my client has lots of pages that display a subscribe form, imagine having a "subscribe to our newsletter block" on every page.

> Aha I understand now - you wish to know about the changes per newsletter subscription. The current implementation is very similar to an entity revision. True, it doesn't directly store differences, however they can be computed easily - just search backwards for the previous history entity with the same email address and compare the subscriptions field.

You can do that for a single subscriber assuming the data is still complete (I could imagine that you'd have a process that eventually cleans old history records), but it's also slow and requires a query per subscriber.

If you do bulk export/analytics of subscription changes of use cases like this, then that just go trickier:
* new subscriptions in last 7 days
* all subscribers with their subscription date for a given subscription

Sure, the hook is an option and then tracking that data somewhere, but that means duplicating the data and not having a history for it. It seems like the history table is on track to be a very useful resource for a lot of things, but right now, it's just not quite there yet.

> Possibly we could add the subscriber ID, however code would need to allow for it not being set. My guess is that developers will mostly fail to do so😃.

Sure, it could be nullable, but I'm not entirely convinced that the data really should be kept if you *delete* a subscriber (as opposed to not having no subscriptions). At least only with the hash stuff. And the hash could also be stored once on the subscriber and still referenced as a hidden record. Just thinking out lout

> My guess is that developers will mostly fail to do so😃.

I didn't look at the implementation, I assumed this is written automatically in which case developers shouldn't write history records directly anyway.

> my client has lots of pages that display a subscribe form,

Agree and this is point 1) from the IS

> Sure, the hook is an option and then tracking that data somewhere, but that means duplicating the data and not having a history for it.

I'm open to adding an extra field with "changes". I guess it could be one field for subscribes and another for unsubscribes??

> but I'm not entirely convinced that the data really should be kept if you *delete* a subscriber

I agree for anon subscribers. They are not normally deleted, instead the subscriptions get removed.

However currently if you delete a user, then the subscriber is automatically deleted. I guess we don't want to delete the history in this case, as we lose proof of consent. Maybe we could change the code to keep the subscriber and remove the subscriptions although I have some doubts.

> My guess is that developers will mostly fail to do so😃.
What I mean is this. Imagine that we add subscriber ID to the history and it can be blank. Developers writing code that reads the history will often not check.

@Berdir I updated the IS that I think matches most of your requests. Did you write any code for your customer in the end, and if so are you willing to share it please?

I added this to the roadmap as I agree it's important.

I'm not entirely convinced that the data really should be kept if you *delete* a subscriber (as opposed to not having no subscriptions).

Interesting point. Already in v3 the code will keep data forever after an anon subscriber unsubscribes, which is probably against legislation such as GDPR. You are correct that in v3 an admin could delete the subscriber from the UI, however I guess that mostly they don't.

Probably we want some config options: how long to keep subscribers; how long to keep history; how long before anonymising history using a hash. I feel that this all seems better in a separate issue, as this one already contains quite a bit.