Requirements and strategy for Core Mailer

I made a first draft to start the discussion.

Here are the old requirements from the original IS. They are mostly outdated by the new IS, however they are kept here because they add some extra detail.

Requirement: extra data associated with each email

• Tag/category that identifies the emails, corresponding to module/key in Core.
• Parameters that define the email to build, according to its tag/category. Matches params in Core.
• The language for the email = langcode in Core.
• Theme to use for templates and CSS. Can be set globally in mailsystem. Adding it here is more flexible and simpler (all the data is available on a single interface).
• User account. Set in Contrib module simplenews, and relevant when rendering entities. For security, recommend switching to anonymous if no specific user is requested to ensure that the access of the current admin user is not used when sending to an ordinary user.
• Transport. Can be set per-category in mailsystem. Add it here for same reason as theme.
• CSS libraries to attach?? Hard-coded in DSM-L. DSM+ allows modules to add libraries with their own email CSS. We could instead get them to add CSS to the single hard-coded library.

These can be get/set by any events/processors and later events can override the value set earlier. Tag is supplied when creating an email and is read-only after that. Language, theme and account are read-write during an initialisation phase, then are read-only once email building begins.

Requirement: building in a callback/event

The code that sends an email should specify just the tag and params. The actual building then occurs in a callback or event. Other code can register events/callbacks to alter the email, or could even replace the entire email building code. This corresponds to hook_mail() and hook_mail_alter() on the existing mail system, and the reasons for doing it are the same as before.

1. Customisability: the code that alters the email has exactly the same information available as the original building code. If the building code outputs a fully-built email directly, then it can be difficult or impossible to reverse-engineer the parameters.
2. Context switching: after the initialisation phase, a context switch occurs of 4 parts: render context (to avoid leaking into current request), language, theme, and user account (for access control and correct rendering of entities). Render switching occurs already in Core and theme switching is done by mailsystem. Language switching occurs in Commerce and is highly advantageous to avoid bugs where the caller forgets to check the langcode. It would allow removal of half the code in user_mail(). User switching occurs in simplenews.

Requirement: flexible and customisable HTML building

The requirement is the same as when building an HTML web-page. The body should be built in a "structured" format (rather than just flat HTML), containing placeholders for values that can be substituted. Later events can therefore alter just one part of the email, and can use placeholders in their replacement HTML. Code can also alter the values of the placeholders. We would expect to re-use the mechanisms we already have: render arrays, TWIG templates/variables, and tokens.

This roughly corresponds to body on Core MailInterface, which is an array that contains a mix of HTML and plain text strings.

Some other fields should support placeholders that can be substituted, especially subject and plain-text body.

Requirement: Extra steps in the email building process

Apart from the ones that require API changes, these could be moved to Contrib. However in some cases the mail system would hardly be useable.

1. Attachment access checking (some done in DSM-L). Important for security, because emailing an attachment has the potential to bypass normal access-checking on the file. This is not just theoretical as there was already a security issue in a popular contrib module. Requires an API.
2. Set some defaults (currently in MailManager)
3. Convert relative URLs to absolute (currently in MailManager)
4. Ensure that subject is plain text (currently in MailManager) - requires a different API from symfony function subject(string)
5. Convert HTML body to plain-text (currently done in DSM-L) - Symfony will do it, but in quite a different way from Drupal
6. Inline CSS (currently done in DSM-L)

Requirement: extensive customisability

1. Allow replacing every part of the Core email building process. This could be achieved by using services, plug-ins, and events (see next bullet).
2. Support flexible registration of callbacks/events. Code can specify the precise timing (for example by means of weights) and the scope (option to restrict to specific tags). Allow multiple registration for different timings/scope. Allow custom code to remove the registrations of other events.

Added a very rough initial draft for a proposal.

I like the idea of a plug-in and a config entity.

I suggest that the plug-in needs full control of the email building process - including to, langcode, etc. So it would be

$template->send($account);
// OR
$email = $template->render($account);
$mailer->send($email);
// OR
$template->render($account)->send();

The advantages are: all the code is in one place; there might be multiple call sites, and we don't want to duplicate logic to each; code that overrides the plug-in or alters the emails can override/alter all the parts - otherwise the to address and langcode are effectively locked; also the to address may depend on language (e.g. send to the site address and site name, the latter being translatable) so it should be evaluated after the language switch.

Other email template types would have different parameters. Contact messages would need $template->send(MessageInterface $message). Simplenews needs 2 params, the issue and the subscriber. This suggests that we need an interface that informs the calling code what parameters to pass - there was clear demand for this in DSM+, and on other issues linked from the META. The interface is identical for all user sub-types and the implementation likely is too, but the config of course is different.

The mechanism of EmailTemplate::render() cannot support this variation in the interface (it can accept arbitrary extra parameters, but it cannot support type information => IDE code completion). This suggests instead the possibility that the plug-in is also a service, which allows all the flexibility of auto-wiring, dependency injection, auto-configuration (to set the plug-in manager automatically to be the factory for the service) etc. We can put the burden of loading the config entity into the framework code rather than requiring each call site to do it - which is important as I believe there won't be universal agreement on the configuration structure: Core will have a basic one and Config may need something more powerful. So the call site code can look like this:

__construct(protected readonly UserMailerInterface $userMailer){}
$this->userMailer->send('password_reset', $account);

or

Drupal::service(UserMailerInterface::class)->send('password_reset', $account);

Note that the interface is the same as the existing _user_mail_notify(), which seems like a sign we are on the right track, and should help during the transition. Once the new mailer is no longer experimental, we could alter all call sites to call Drupal::service(UserMailerInterface::class)->send. Then the new mailer plug-in could check if the site has chosen to enable the new mailer service yet, and if not instead call _user_mail_notify().

The proposed resolution covers only some of the acceptance criteria. Others parts are covered well in some existing issues:

• ✨ Create new mailer service based on symfony Active
• ✨ Create a new mailer service to wrap symfony mailer Active
• ✨ Events/callbacks for new mailer service Active

Filed two issues, both of them with draft MR which very roughly illustrate the approach:

• 📌 Introduce EmailTemplate config entity and plugin type Active
• 📌 Use EmailTemplate config entities in core modules Active
  This illustrates the core call site, conversion from hook_mail to MailTemplate plugin code and also how the experimental mailer could be turned on and off on a per module basis using container parameters.

Filed another issue for the context switching. This approach could help in other cases outside the mail system as well.

• ✨ Execution environment: Safely run code inside an environment with partially substituted system state / config Active

Regarding #9, parameter typing is an important aspect of the design and that merits a thorough discussion. I think there are two perspectives on this which partly overlap and partly contradict each other.

• The perspective of the developer: It is desirable that the IDE is capable of inferring types from the context, providing help and docs, and also highlighting errors. This is possible if the code has a good level of type hinting in place. Also the quality of reports by static analysis (phpstan and phpcs) gets better the more type information is available.
• The perspective of a site builder: It is desirable that information on data structures are shared across subsystems. E.g., when building a view it is crucial that the list of fields on a bundle of an entity is available. This is another kind of typing information which has no relation to PHP type hints. Config schemas and the TypedData API provide the necessary infrastructure to make this work.

I opted for the TypedData approach in #11. I think this could provide the necessary metadata such that it is easier for other modules (e.g., ECA) to interact safely with mail templates.

Regrettably I haven't come across an easy way to make the TypedData information available in the IDE. To some extent it could be possible using generics, but that just annotations for now and it isn't a language feature in PHP (yet).

Trying to take a birds eye look at the architecture in full context (i.e. with 📌 Use EmailTemplate config entities in core modules Active and 📌 Introduce EmailTemplate config entity and plugin type Active ) applied). I think the combination of EmailTemplate config and EmailTemplate plugin works out.

I'm not happy with the EmailType YAML plugin though. It currently serves as a discriminator on the template config as well as on the template plugin. An UI could filter the available plugins matching their data type to an existing config when the user wants to switch to another template implementation. However, that could be easily done with TypedData as well.

The only unique responsibility of the EmailType at the moment is the default_template key in combination with EmailTemplate::onDependencyRemoval() implementation. This is done for the following reason:

Reset the template plugin to the default if the template plugin has been customized and if the provider (module) of the template plugin is going to be removed.

It might be better to solve this in a different way. E.g., block plugins use a broken plugin as a fallback. This could also help in the case where an email template has been switched to a third-party implementation which was removed subsequently.

Like this it would be possible to remove EmailType, EmailTypeInterface, EmailTypePluginManager, EmailTypePluginManagerInterface. Also email sending modules wouldn't need to add an MODULE_NAME.email_types.yml anymore. That would reduce the boilerplate quite a bit I guess.

Like this it would be possible to remove EmailType, EmailTypeInterface, EmailTypePluginManager, EmailTypePluginManagerInterface. Also email sending modules wouldn't need to add an MODULE_NAME.email_types.yml anymore. That would reduce the boilerplate quite a bit I guess.

Pushed that to the referenced issues. In my opinion this looks cleaner.

I already mentioned in #9 that I believe there won't be universal agreement on the configuration structure: Core will have a basic one and Config may need something more powerful.

The 3rd paragraph of the IS states a clear scope which excludes configuration, suggesting instead that the next step is the
"Central Mailer". That's not really a good name, probably it would better be "Drupal adaption layer" - continuing upward from transport and delivery layers that we already have. The main content is the issues in #10, which are still awaiting response. The plug-ins and configuration would sit above this.

We discussed this during a call and by agreement we updated the IS to remove the statement of scope from #16.

I have updated the IS to give each requirement a code and a short name so we can reference them in the comments. Obviously this will only work if we avoid changing them😃.

Personally speaking I still feel that the "Drupal adaption layer" (I'll abbreviate to DAL) is the place to work next because it is the foundation for everything that comes after. We are working up the stack layer by layer.

I'll try to describe it here. We already have the delivery layer which has interface Symfony\Component\Mailer\MailerInterface. There is a single function send(RawMessage $message, ?Envelope $envelope = null).

My idea is that the DAL provides an interface with a very similar feel.

• Drupal EmailInterface which enhances the Symfony Email.php with Drupal information
• Drupal service Mailer class implementing MailerInterface
• Single function send(EmailInterface $email)

It covers the following requirements:

• [D4 EMAIL_INFO] by creating EmailInterface
• [B3 LANGUAGE_CONTEXT], [B4 THEME_CONTEXT], [B5 USER_CONTEXT], [C2 SAFE_ACCESS] as the service does context switching
• [C1 SAFE_ATTACH] as the service provides an API, although this might come after the beta
• [D3 CAN_ALTER] as the service will fire events
• [D5 STRUCTURED_BODY] as the EmailInterface will have get/set methods for accessing a render array.

So it does a lot, yet it is fairly compact (1000 lines of code total?) and can be split into stages, each one a clearly scoped issue. I feel that it is less controversial than the layers that come above, and we have a chance to get agreement in a reasonable time.

I had a look at 📌 Introduce EmailTemplate config entity and plugin type Active and 📌 Use EmailTemplate config entities in core modules Active . There are a lot of really interesting and ingenious ideas.

My first impression is that it's much too complex especially in comparison with DSM+ yet fewer features (e.g. still only plain-text).

DSM+ v2.x has a class UserMailer which has 57 lines of code, which includes HTML support but excludes the actual configuration of subject and body.

In 📌 Use EmailTemplate config entities in core modules Active there are

• 9 config files of 15 lines each
• 300 lines of PHP split across 4 new files
• 80 lines of code in _user_mail_notify() which is code that presumably fails to meet the requirement [D3 CAN_ALTER].

So 10 times as many lines, and 20 times as many files.

IMHO, the EmailTemplate config entity isn't the right direction. [D1 METADATA] can simply and naturally be solved by putting the email sub-types into the annotation for the plug-in. This is a standard pattern that we see many times throughout Core and Contrib. Code can alter the plug-in definitions and implementations using an info_alter() hook.

AFAICS the EmailTemplate config entity contains two parts:

• Allow altering of plugin metadata info using config
• Allow replacing of plugin implementation using config

I don't yet see a clear requirement for this added complexity, which seems non-standard for Core. It's also less robust as email sending would break if the config is missing.

Catching up on the discussion and merge requests. Long comment incoming...

I don't think LOC is the primary factor to consider for this. Also because the current examples are way more complicated than they need to be due to feature flags, the experimental module and BC. The actual implementation would be a lot less.

That said, I do agree that the config entity approach seems complex. Yes, config entity + plugin are a very common pattern, but sometimes also overused. This feels similar to action plugins and config entities, which are often pretty awkward as it's typically a 1:1 relationship and tedious for modules to maintain the config. In general, code that is hardcoding specific config entity ID's is a bit of a code smell to me. As @adamps said, it's unclear what would happen if that config is missing. For example now with recipes, which pretty much *encourages* installing modules without their config ( I don't really agree with that, but that's another topic). Deleting the password reset template

Naming: I think the template part is misleading. That makes me think of twig templates, HTML and customizing their output. Looking at DSM, it uses build() instead of render(), which makes a lot more sense to me. We build an Email object, which might or might not involve rendering something or using templates, but it's not the main thing. It's also similar to terminology used in simplenews.

An issue with the current architecture is that there's a bidirectional dependency between the plugin and the config entity, as it's passed to render. If you compare with block plugins for example, they don't know about block.module block config entities and can be used in other contexts as well. Similar for field formatters, which are also used by views. That's what $configuration is for, the config entity is responsible for storing and passing that in.

Now, there are cases where a fully implemented config entity + plugin could be useful. For example with user, the plugin would be provide a subject and body setting which would make it self-contained instead of accessing global config (which currently bypasses your attempt at making not just mails but also their data and arguments documented and discoverable). and status is built-in, so the code could check that. But it would mean that the UI would move from user settings to generic mail configuration, which is likely harder to find. It's also not as flexible as you'd be used to from plugins. you can't make up your own notifications, you can only enable/disable and edit the specific settings of them. In the end, pretty limited benefit over the current config and some drawbacks.

contact, update are examples that don't really benefit from there being a config entity, and that's going to be a lot of contrib and custom examples. That's combined with the "problem" that the mail rendering/preparing sits entirely on top of the actual mail send API and is there fore optional. People looking at this and trying to send mails will just directly create the Email object and send it directly. And if they do that, we lose the ability to use a mail theme because rendering of stuff possibly already happened. I think we should make it as easy as possible to use the mail API without losing features like language. And then optional features on top. If you look at commerce_mail(), they essentially bypass the whole thing by just passing in the body and rendering it there. I wonder if we should actually allow to use a callback based approach for those minimal cases, similar to what renderInContext() essentially already is.

I'm not sure how important discoverability and reusability of mail "templates" really is, it might in some cases even be undesired. We recently discussed that ECA explicitly doesn't support those special user password reset mails, to not allow to send them out in the wrong context, that could even be a security issue.

On the other side of the spectrum, use cases that actually have generic and complex use cases around mails might struggle to fit into this architecture. Simplenews has multiple things that make up a newsletter, specifically a entity that sent to a subscriber and there's a lot of rendering and logic going on that should happen within the mail render context. And ECA would need to redefine its mails that sit in its configuration or have a super generic one that would almost become recursive.

I think I get the intent with the types and data, but that really feels complex to use. the user confirm thing requires a data type plugin, a definition class, it's passed in as a raw value, converted to typed data and then used. It values (theoretical) reusability over DX, despite all that complexity, what you work with when writing the code is an array with arbitrary keys. I see that DSM has the concept of mail params on the object. That's still keys and not types that can be enforced, but I think that's an improvement over the single magic mixed type that this has. With mulitple params, we can still document them and their type.

I've even been thinking about going entirely with plain value objects implementing a basic interface, but haven't fully thought that through. But overall I'd clearly recommend keeping the actual mail builder interface/API as plain and un-opinionated as possible and start with that.

Thanks @berdir

I don't think LOC is the primary factor to consider for this. Also because the current examples are way more complicated than they need to be due to feature flags, the experimental module and BC. The actual implementation would be a lot less.

I accept all that and I certainly don't believe we should be trying to minimise lines of code in an artificial way. I mentioned LOC just as an illustration - I could equally well present a list of detailed specific points to demonstrate my point about complexity.

But overall I'd clearly recommend keeping the actual mail builder interface/API as plain and un-opinionated as possible and start with that.

I certainly agree with that and it's exactly what I was trying to do with #19.

I started developing DSM+ 4 years ago to meet very similar same user stories and requirements. Through the experience of 46k sites we fixed 313 issues. Various other Contrib modules and likely numerous sites have written code to integrate with the interfaces so if we can keep those interfaces it would simplify migration of these sites to Core. Recently the 2.x branch is a significant rewrite based on all the things we learnt in 1.x. I agree we shouldn't blindly copy it into Core, but also we shouldn't blindly ignore it. From my perspective, before prototyping a new system we should look at the existing one and see what we feel needs to change and why.

Certainly @znerol has some ingenious ideas which we can also include in our overall solution. Also I see some things that I tried 4 years ago and didn't seem to work out so well; I see some key points that I learnt through direct user feedback that he likely would also need to consider. This is entirely natural as DSM+ is a mature codebase proven in the field in comparison with a prototype developed over a few weeks (or maybe months) by one talented developer.

Thank a lot @berdir for the review.

Coming out of discussions at our last meeting I have been contemplating how we can progress efficiently with group agreement.

I feel that we will move forward by making individual decisions. Each one is a specific choice in a single localised area from a defined set of options. First we make a choice regarding the existence of an EmailInterface, which if we agree, leads to an issue. Then within that issue we have separate choices about each of the fields that could be on it, leading to an actionable issue that can be coded (the coder can make any more detailed choices themselves). Probably we will make 100+ of these choices together as a group. It seems a lot, however many of them can be done in 2 minutes. I felt that @zengenuity mostly agrees (please correct me if I'm wrong).

Lorenz says that he likes to think in code. I feel it would work well to make prototypes within one of the issues that we agreed as a group. We could have an issue for creating the component mail building code, and we could agree that for each idea someone had they would create a MR for only user module - maybe just 50 or 100 LOC. The author could explain the key points and advantages in a comment.

However prototypes for the large parts of the system (500-1000 LOC) from my point of view are not good for making decisions because it is like making 50 choices at once, and it's too much to take in. Potentially they generate multiple issues = noise in the issue queues, sometimes duplicating other issues, and drawing the attention of outside reviewers to the prototype rather than the decisions. Personally speaking I wonder if this kind of prototyping might even be better on a personal gitlab clone or similar??

How does anyone else feel?

Older version saved for reference

Proposed resolution

Introduce a new plugin type and a config entity type which work in tandem. I.e., use plugins to realize a strategy pattern (substitutable implementations) and the config entity to select one of them for each case. The combination of plugin type and config entity type is a known pattern in Drupal core. Examples for this architecture are Action, Block and Editor.

A sensible name for the plugin type and config entity in the mailer realm could be EmailTemplate.

The parts should roughly fit together as follows:

• Each implementation (email template plugin) declares which type of email it is capable of rendering. The responsibility of an email template plugin is to turn input data of a specified type into an email data object.
• Each call site (email sending code) declares the type of email and its specific variation it wishes to dispatch.
• A config (email template entity) links the email type+variation combination to a specific template implementation.

Call site

The call site (email sending code) interacts with the email template config entity. Config entities are easy to deal with in both procedural and object oriented code. The render() call is forwarded to the plugin implementation.

Procedural example (error handling removed):

$account = User::load(1);
$template = EmailTemplate::load('user.password_reset');
$email = $template->render($account, $account->getPreferredLangcode());
$mailer->send($email->to($account->getEmail()));

EmailTemplate config entity

Config entities can track their dependencies. Config which is shipped in a config/optional directory is checked whenever a module is enabled or uninstalled. Optional config is installed as soon as all of its dependencies are met and removed whenever one of its dependencies breaks. It follows that the experimental mailer module can safely provide config for all mail sending core modules. Whenever the experimental mailer module is enabled, the subset of email template config entities having all dependencies met is installed. Whenever the experimental mailer module is disabled, all the email template config entities are removed and the system is config wise back to the point before enabling the mailer module.

Contrib and custom code can list the email template config entities. This represents an exhaustive list of all transactional mails potentially sent out from a site.

EmailTemplate plugin

Core may initially ship rudimentary mail template plugins mirroring the current behavior of core hook_mail implementations. Better implementations (e.g. based on entity view modes) can be added later in the development process.

Contrib mail sending modules (e.g., Commerce, ECA, Simplenews, Webform) can add their own mail template plugins and config entities for their own mail types.

Contrib mail building modules (e.g., DSM+, Easy Email) can ship additional mail template plugins for existing mail types and substitute the default implementations.

I copied the overall design in from the meeting notes and tidied it up slightly. Next step @zengenuity and @adamps to develop ready for review.

Some thoughts on the updated proposal and comments:

I agree that code is useful to experiment and review the direction and it is specifically also useful to see the bigger picture, not just one single decision/class/... But I'd essentially try to do a "vertical prototype" in this phase, convert 1-3 examples, e.g. one fixed mail like update and one for user or something like that. A lot of work was put into defining every single config entity for user e-mails which wasn't really necessary to understand the concept.

Just like with the prototype, I'd avoid creating too many issues to discuss and implement every single bit. It's hard to get things into core piecemeal. And a single issue (fork) can fit many different merge requests, if different people want to experiment/work on different parts separately.

IMHO, the hardest thing of the whole process is experimental code/module and BC in all kinds of directions. Usually with experimental modules, we keep them everything isolated to/in them. The plugin/config entity MR had to make changes to lots of modules, with feature flags and module checks, but still. AFAIK experimental modules still get removed form releases, that will be very hard to handle then. It's probably not realistic that this is beta stability by the time we ship 11.3. We need to have answers for this, both for core while this is experimental as well as contrib, which might want to start adopting this without breaking compatibility with earlier versions.

Rough ideas on that:
* Instead of per module, I think we should have a single flag that switches all (supported) mails to the new API. maybe the experimental module can be that flag and that's enough, or we have something else. It seems unlikely that sites want to switch user mails but not contact for example. If we allow sites to decide, then we might be able to get away to not have bidirectional BC support for something like hook_mail_alter (which sounds like a nightmare)
* Not much we can do about modules that don't support this yet. Don't think we want or can automatically transform hook_mail() implementations, there are too many custom things going on there with attachments, HTML and what not. However, we could adopt something that helps sites identify modules that aren't converted yet. Inspired by OOP hook conversion, we could maybe support an attribute on hook_mail() / alter implementations like `#[SupportsMailer]`, then we can check if a site has any modules installed that don't have this yet. They'll still work, but sites will need to be aware to configure their mail transports in two places and so on.
* Contact was/is interesting in the MR because it's a separate service. Instead of replacing that conditionally in contact, we can do that in the module. It might be beneficial to push _user_mail_notify() into a service to do the same thing there as well. Then mailer could provide alternative services for all core modules that mails. And once it's no longer experimental/part of core, we move it back.

Oops, restoring the issue summary.

Two more things:

* I think I saw 3-4 similar/older issues like this one just when searching for _user_mail_notify, there are likely more. We should close some old plan/meta issues or relate them if they're still relevant. Like ✨ Create a plug-in system to build emails with Symfony Mailer Active .
* As mentioned to @znerol, the current meeting slot really doesn't work well for me, due to work travel/family. Calls can be useful for specific discussions, but other initiatives have adopted async meetings (e.g. discussion threads over 24h or so), which I think would help getting people from other timezones into this as well. I also like to have some time to collect and order my thoughts (and then write very long issue comments), I usually struggle to provide meaningful feedback in calls.

I found #3229038: Consolidate public APIs for generating user action URLs into a service → but that doesn't seem to touch _user_mail_notify() therefore filed 📌 Extract _user_mail_notify() into a user MailHandler Active .

I added that to 🌱 Mailer module roadmap: the path to beta and stable Active . It seems straight forward and actionable.

I feel that we should have a group decision process for agreeing which issues are actionable. At very least it could be like setting RTBC status - please don't put your own issue in, at least one other person should be involved😃. FWIW I would put this issue in the "Mail building Components" section in the roadmap.

Personally speaking, I definitely agree with the idea. We are currently having a little discussion in the issue about how to do it, then once we have agreement at that level it becomes actionable.

FWIW I would put this issue in the "Mail building Components" section in the roadmap.

I'm okay with that, just do it.

IMHO, the hardest thing of the whole process is experimental code/module and BC in all kinds of directions.

Yes you are right, we should have some requirements about that.

They'll still work, but sites will need to be aware to configure their mail transports in two places and so on.

That sounds scary to get working correctly, to test and support. Maybe we should allow only one active mail system on a site?? Even so it will be plenty hard enough.

I think we should have a single flag that switches all (supported) mails to the new API.

I was thinking the same thing. Let's keep it simple.

Don't think we want or can automatically transform hook_mail() implementations, there are too many custom things going on there with attachments, HTML and what not.

Actually it is definitely possible without too much complexity, we have it in DSM+. Not necessarily guaranteed to work 100%, but in practice we haven't really had new issues recently. Perhaps it doesn't need to be in Core - sites could choose to enable it from Contrib if they want to work that way.

So perhaps the requirements overall are something like this:

1. The experimental mailer reports which modules support which mail systems (some might support only one, some both).
2. The experimental mailer module provides a setting to choose which mail system is active: old or new. All modules use the active mail system. (We could skip this page at first, and use enabling the experimental mailer as the switch. However it's arguably a more realistic beta if we have the page, and it allows the user to see the level of module support before switching)
3. By default, emails are dropped if the module doesn't support the active mail system. However there is a mechanism to enable a Contrib module to perform conversion.

I'm not sure how important discoverability and reusability of mail "templates" really is, it might in some cases even be undesired. We recently discussed that ECA explicitly doesn't support those special user password reset mails, to not allow to send them out in the wrong context, that could even be a security issue.

Interesting point. We definitely do need the discoverability as modules like DSM+ "Mailer Policy" need to know there is a password reset email so they can show a form to configure the subject and body. Removing the discoverability won't help security anyway: we'd end up instead with a form that allows the admin to type in the ID of the tag/key (so they type password_reset), and it would be equally insecure. Or the admin could apply the hijacking context setting globally to all user emails, or even to all emails. Or the admin could enable an email logging module and read the logs, or they could configure a transport that sends emails to their own private server, or enable a module that redirects all emails (normally used on a test site) to a different address.

During the meeting we added a new requirement C3 to cover this case, which I imagine would work a bit like this. The metadata for each email sending component can optionally include a setting to indicate a specific email category/tag is sensitive. The Email can have a corresponding isSensitive() function. We can create a Core permission "view sensitive emails". The functions that would cause a problem including setBcc(), setTo(), setCc(), getBody() can then all perform a check on the permission if the email is sensitive. I guess we need a setting for code to disable the check in case of trusted code that isn't related to a UI. We don't necessarily need all this for a beta however.

I'd avoid creating too many issues to discuss and implement every single bit. It's hard to get things into core piecemeal.

OK thanks for the advice. Still, I feel we need a way to take some decisions individually (and sure we could change our minds later), rather than evaluating the whole thing in a single MR.

I see that DSM has the concept of mail params on the object. That's still keys and not types that can be enforced, but I think that's an improvement over the single magic mixed type that this has. With mulitple params, we can still document them and their type.

Based on feedback from 1.x, now in v2.x we have a clear interface such as UserMailerInterface which has types for each function argument. These then get converted into the type-less array params, but we have at least ensured they are the right type in the first place. We also have a function setEntityParam(EntityInterface $entity) which automatically sets the param with a key matching the entity type, so it gives some level of type hinting. We could even add add code that prevents overwriting such a param with the wrong type.

In addition to the initial params set at the call site, we found that hook code sometimes just wants to record some data into params (perhaps to read again at a later stage in the email pipeline). Forms have setValue(), rendering allows adding anything to $variables. So probably we can't reasonably try to control everything that goes into params.

We should close some old plan/meta issues or relate them if they're still relevant.

I feel we do currently have a problem with multiplying and semi-duplicating issues. I raised a bunch of issues a year or two ago. Some of them are a little out of date, but still they have many interesting ideas, and also some valuable discussions with @berdir and other core maintainers.

@znerol has since raised his own issues. They tend to operate in a different "space" without referencing the prior ones, often using different terminology. Some of them might now be outdated, others still contain important ideas or MR.

I feel that it would help now to merge into a single set of issues referenced from this IS that we all use moving forward. We can transfer information in from others and then close them as duplicates. Once the group has agreed that an issue is in scope, then we can open it up for coding and allow different MR to be compared side-by-side.

As per #29, @zengenuity and @adamps will aim to create a design and propose how it could map to issues, bearing in mind @berdir feedback. Then we can get review from the group, and if we manage to get agreement someone can do all the issue management.

I started working on a new approach for the core mail building infrastructure: 📌 Introduce email yaml plugin and an email building pattern known from controllers Active

I think this should feel quite familiar for people who are used to implement their own controllers.

In this iteration I also put more effort in the HTML/templating part. I took a look at the entity templates (node.html.twig, taxonomy-term.html.twig, media.html.twig, etc.). All of them use a contents variable containing a list of content items. I think it would be good to use the same pattern in email.html.twig.

This is literally a beer idea, curtsy of @jurgenhaas → . I'm recording this here just for completeness:

If we think 📌 Introduce email yaml plugin and an email building pattern known from controllers Active one step further, email could just be one of the supported response formats. In order to render an email, create a Request object with appropriate attributes, dispatch it through the http kernel as a subrequest, grab the email object from the response and send it via the transport framework to wherever its needed.

On the way through the request / response cycle, parameter upcasting, language negotiation, theme negotiation, render context isolation, render caching and maybe even cooperative multitasking via fibers is performed just like with every other request. Without experimentation its impossible to know what works and what breaks, though.

It would also be interesting to explore whether or not this code flow would simplify access checking on embedded attachments and images.

Considering email builders as routes (or microservices, if you prefer) also raises interesting questions. For example, some email builders likely are idempotent (simple notifications are always the same when generated with the same input) while others are not (the password reset link must be newly generated).

Without experimentation its impossible to know what works and what breaks, though.

We have one 4 year experiment including over 40k sites that is known to work😃.

In 📌 Introduce email yaml plugin and an email building pattern known from controllers Active :

The rendering itself happens inside an EmailRenderer service which has the following responsibilities:

That sounds rather like ✨ Create a new mailer service to wrap symfony mailer Active :

The Email building pipeline proceeds in the following sequence:

Also I see you are introducing template suggestions, email plugin manager, some metadata on the plugin definition, and various other things quite similar to DSM+. You have some really interesting new ideas that I like. So that is quite promising. On the other hand, I can see many ways in which DSM+ is fairly obviously better, and the MR code would fail to meet needs that have been clearly expressed by multiple developers in Contrib issues.

Many of my comments remain unchanged from before. I find these huge MR too big to process in any sensible way. There seems to be a danger in "reinventing the wheel". We should also consider simplifying the transition process for the existing sites, so try to avoid replacing an interface with something mostly equivalent, when perhaps we could leave it the same and it be equally good.

I feel that likely you experiment with these big prototypes to clarify your own thinking, and to explore radical new options. Which is fine, but eventually we need to find our way to agreement on issues we could commit.

Here are some steps that I would see as useful:

• Start discussion on the "Drupal adaption layer" to create a Drupal Mailer service and EmailInterface. These map very clearly to many of the agreed requirements, and I feel are essential. It would significantly simplify all the MR you create, because currently you have a lot of "boiler plate" code that could be handled by this layer.
• Work on details in specific issues with a smaller scope. This would help me even if @berdir is right and we have to bring it back together to get a commit. Obviously this requires some agreement on a design, which I am keen to discuss.
• Provide a clear explanation for the choices made, especially when it is re-inventing very similar proven Contrib code. If you could reference a DSM+ class/concept/etc and specify 3 improvements that you would like to make then I can likely easily agree. I would then point out 5 things you apparently made worse, you might agree in 4 cases, it was just an accident, we discuss the 5th case, and very quickly we are done.
• Reuse or reference existing issues and ideas from other people rather than duplication.

I added some new requirements based on recent comments and some of my own ideas, all marked DRAFT. I propose that one existing requirement be deleted, marked DELETE.

Please comment to say whether you agree, or alternatively we could do it at our next meeting.

@zengenuity and I had a useful discussion of the design. We tended to agree at the high level, then had different terminology and ideas at the detailed level. I adjusted the IS proposed resolution slightly, including to reverse the order of presentation (sorry!) because this will match the order we work in - continuing our journey up the layers, each one dependent on the ones below. The next layer is ...

3) The "Drupal adaptation layer". We already have detailed issues with a list of points for discussion in the proposed resolution. Following @berdir suggestion I feel we should combine them into one (which it was originally!). ✨ Create new mailer service based on symfony Active , ✨ Create a new mailer service to wrap symfony mailer Active , ✨ Events/callbacks for new mailer service Active .

4) Then I propose that the final layer for Beta would be the building layer.

We can leave the others....

5) In the previous meeting and in a previous prototype I recall that we had the idea that for the Beta we could stick with the existing config layer. So Core generates plain text mails, and Contrib can override for HTML. I have kept that idea as a proposal in the IS.

6) @znerol already prototyped how we can avoid changing the user layer for beta, by overriding the old mail system services. I added that to the IS, it seems clearly good.

Before we move on to work on the individual layers, please can people comment on how they feel about the overall picture of the layers, and what we would do in our first beta version??

AFAICS 📌 Introduce email yaml plugin and an email building pattern known from controllers Active covers layers 3 to 5.

If we implement layer 3 (Drupal Adaptation) it would likely take in EmailRenderer. It would simplify the rest of the code quite a bit. We could remove all reference to $langcode because the language would already be switched. We could integrate concepts like token replacement into the underlying service.

The yaml plugin is a really interesting idea that I would like to discuss and I believe it could work well. I find it difficult to discuss in the context of 📌 Introduce email yaml plugin and an email building pattern known from controllers Active because that contains a lot of details also about 10 or 20 other ideas, some of which I find good, some bad. As I said before, I propose we will make progress by taking decisions one-by-one.

I feel that the Mailer service from ✨ Create a new mailer service to wrap symfony mailer Active is a good match with the yaml plugin. The Mailer interface could be written as

function send($id, $params)

And the call-site code then becomes attractively simple

class UserNotificationHandler {
  public function sendRegisterAdminCreated(UserInterface $user): bool {
    return $this->mailer->send('user.register_admin_created', ['user' => $user]);
  }
}

All the work is now in the "controller" part, the mail builder, or whatever we call it. It's easier to understand in one place, and it's easier to swap the implementation (compared with the old mail system, where part of the code is in _user_mail_notify(), part is in user_mail() and neither can be swapped!).

We could even add parameters to the yaml file

    parameters:
      user:
        type: entity:user

although perhaps even better we could automatically deduce the type because the name is an entity type provided by the same module.

The $langcode is tricky and indeed merits an in-depth discussion. I think it is worthwhile to consider it thoroughly whether it should remain a required parameter (like it is today, and like it was on all my prototypes) or whether it should be dropped (as suggested by some comments). And if we decide to drop $langcode from the call-site method signature, we should discuss that further with language subsystem maintainers I guess.

From the top of my head, I can think of at least four ways to determine the language for an outgoing mail:

1. Use the site default language.
2. Use the negotiated language (this only works if the mail sending is triggered in response to a browser request, it falls back to 1 on cron runs).
3. Use the recipients preferred language (if the message has only one recipient and it is linked to a known user account).
4. Use a language stored on a related entity (e.g., the language could be stored on a newsletter subscription).

It seems to me that the choice of language is context dependent. This could be an indicator that indeed the call-site is the most appropriate location to set the langcode. At the call-site, the negotiated language is still available (via ConfigurableLanguageManager::getCurrentLanguage()). Depending on how the context switching is implemented, the negotiated language might return a different result inside the mail builder (i.e., inside the switched context). That in turn means, that mail builders have less options to choose from when attempting to determine the correct message language.

The call-site also might be aware of whether or not it is running in cron (update) or from within a form submission. Issues like 🐛 Password reset invalid mail notify language Needs work also indicate, that some users expectations are different than what others find intuitive.

Keeping the $langcode on the call-site method signature forces developers to make a sensible choice. It is for sure debatable whether or not this is a good thing.

On the other hand, neither the old interface nor any of my prototypes has references to other i18n information. E.g., for certain messages it would be important to know the recipients timezone (e.g., calendar invites). So why do we have $langcode as a required parameter on the call-site method signature but not $timezone?