Determine some heuristics for feature inclusion to avoid conflicts with core and difficult upgrade paths for sites

Created on 17 May 2024, 11 months ago

Problem/Motivation

I think we need a discussion about whether and how to include modules that are possibly going to be (re-)implemented in core in the medium term, to avoid duplication of effort and making things difficult for site-owners unnecessarily.

This is not about deciding what features are desirable, but more about how to include those features if there is not a clear single solution. i.e. this issue isn't talking about pathauto or webform where there are clear stand-out solutions that if they were implemented in core would be via moving those modules into core, not a re-implementation.

Because we're doing a recipe/composer template, we don't have to provide an upgrade path between Drupal CMS versions, which is good.

However, sites that install Drupal CMS, and turn into actual sites, they will need to be updated - to new core minor versions, to new versions of contrib modules, to new major Drupal versions.

When we include 'golden contrib' modules like pathauto, that is fine, because we know they have huge install bases and are well maintained. Sometimes things change, but it can take a long time.

However there are other modules that closely integrate with, or replace, and existing core feature, and we need to think carefully about how to plan for the lifecycle of those modules/features, at least on a 6-12 month timespan. We will face similar, but probably smaller issues, if we're trying to choose between 2-5 similar competing contrib modules too.

Example 1: Layout Builder Restrictions

https://www.drupal.org/project/layout_builder_restrictions β†’

This already has a note on the project page saying:

"Drupal core will soon provide a method for controlling available blocks in Layout Builder. See #3365551: Add the notion of a 'configured layout builder block' to solve a number of content-editor and performance pain points. This core solution should be sufficient for most site needs, so consider using that instead of Layout Builder Restrictions. For advanced use cases, Layout Builder Restrictions may still be useful."

Let's assume 5,000 new installs of starshot per month, and that starts from when it hits 1.0.0 in September. So 15,000 installs by 11.1

If we put layout_builder_restrictions in starshot, and then we fix https://www.drupal.org/project/drupal/issues/3365551 ✨ Add the notion of a 'configured layout builder block' to solve a number of content-editor and performance pain points Active in 11.1, then we've just increased layout_builder_restrictions usage by 1.5 its current level.

There's then 15,000 sites with it installed on top of the 30k there already were, but they didn't intentionally install it, they installed it because it was bundled with starshot - i.e. we installed it for them.

Now 15,000 people need to decide whether they actually need layout_builder_restrictions or not - if they are new low/no-code people they might have no idea. But it doesn't matter hugely at this point because there's no conflict as such, we just made it superfluous for 98% of sites with it installed.

Then in 11.2 we add a permissions-based system for site owners to curate the available layout blocks in core, which has its own UX actively conflicting with layout_builder_restrictions. Now those same 15,000 people are forced to navigate the transition and deal with confusing UX/multiple sources of restrictions.

All that time, we, and the layout_builder_restrictions maintainers, knew that this would happen, and it was advertised on their project page.

All of this vs. the block list being clunky for 2-3 months, then 11.1 trims it down.

Example 2: linkit

https://www.drupal.org/project/linkit β†’

Linkit provides a ckeditor5 link plugin with a lot of features over the core one.

However ✨ Drastically improve Drupal's default linking experience in text fields Needs work has been open for two years to bring some of the main features of linkit into core. It needs a bit more work to be committable, but if we prioritized it, would have a very good chance of landing in 10.4/11.1.

Once that is committed various things are likely:

1. There might need to be a new version of linkit that drops the features core provides and maybe adds back the remaining features differently.

2. the core link plugin might provide enough features that lots of sites don't need linkit at all any more.

3. Sites with linkit installed will (probably) need to make various choices about whether to switch to the core plugin, update to newer versions of linkit etc. This would require at least editor configuration changes.

In this case, again, I think both Drupal CMS and people that download it, would be better off if we fixed the core issue rather than including linkit, then removing it again (leaving tens of thousands of additional sites with a contrib -> core migration to worry about).

3. Gin

The starshot prototype includes Gin, which is a subtheme of Claro with additional features like dark mode.

Gin is implemented as a subtheme of Claro which means it's not a full replacement, but rather an enhancement built on top of Claro.

You can switch from Gin to Claro, and there is no impact on your data model, only your UX and theme configuration.

It is possible that we might want to implement some Gin features (e.g. dark mode) directly into Claro in the future.

While this might have some complexity, for me I don't think it's likely to cause much of an issue compared to layout builder restrictions or linkit because the impact on site configuration will be minimal, and admin theme settings are almost inherently reversible compared to other site configuration (finite parameters, don't affect config entities etc.)

4. Same page preview https://www.drupal.org/project/same_page_preview β†’

This module has been suggested for core ✨ Consider adding Same Page Preview to core Active .

I (catch) haven't reviewed it, but I think for inclusion into Drupal CMS, we should consider 2-3 things:

1. Do we want to replace core's form-based preview button with same page preview? If so, then including the contrib module in starshot could be a good way to collect more feedback on it. If we weren't going to replace core preview with this, then starshot would essentially 'hide' a core feature from a large number of users, which would be a strange situation to have for a long time.

2. Core already includes two other potentially conflicting preview mechanisms - 1. layout builder allows preview of layout overrides/content already, and this could be improved as part of layout builder/experience builder improvements. 2. Workspaces allows 'full site preview' as in when you're in a workspace, you can see the draft versions of content in all situations where you browse the site - canonical entity pages, listings etc. including any related entities like menu items, media etc.

Steps to reproduce

Proposed resolution

Ideally come up with some general heuristics, although a useful discussion that helps us think about individual cases would be a start.

Remaining tasks

More examples and ensuring the current ones are accurate/fleshing them out.

User interface changes

API changes

Data model changes

πŸ“Œ Task
Status

Active

Component

Code

Created by

πŸ‡¬πŸ‡§United Kingdom catch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @catch
  • πŸ‡¬πŸ‡§United Kingdom catch
  • πŸ‡¬πŸ‡§United Kingdom catch
  • πŸ‡¬πŸ‡§United Kingdom catch
  • πŸ‡¦πŸ‡ΊAustralia pameeela

    We have already had conversations about some of the specific cases referenced here, but agree it would nice to have some underlying principles written down.

  • πŸ‡¬πŸ‡§United Kingdom catch

    More questions have come up since I opened this originally.

    If someone installs Drupal CMS 1.0.0, should they expect that all of the modules that come with it are covered by the Drupal security team?
    If so, should they also expect that if they install Drupal CMS 1.0.0-rc1?

    If they install 1.0.0-beta1 (or rc1 for that matter) should they expect to be able to continue building their site on it if they don't run into problems, and then use automatic updates to update the various modules installed?

  • πŸ‡ΊπŸ‡ΈUnited States greggles Denver, Colorado, USA

    Following.

    As far as I know, the Drupal CMS project does not change expectations about what it means when a maintainr opts into security coverage or does/doesn't make a stable releases.

  • πŸ‡¬πŸ‡§United Kingdom catch

    I don't think it changes expectations about the modules, but I think that people installing Drupal CMS will have expectations about security support, or if they don't, would expect to be able to find out what the parameters are.

    Drupal CMS doesn't pin dependencies to a specific release, only major or minor (and IMO can't pin any further than that, because sites are responsible for updating modules after that).

    So in that sense unless there is a security issue in a recipe (wrong permission granted to wrong role or similar), or a zero-day in a module that needs mitigation (removing permissions or a feature), then a security release for a module is only an issue for a site - new Drupal CMS users will get it automatically.

    So then the main risk for a module that is on an non-release branch or not opted into security support is that a security issue found would be opened in the public queue.

    For modules with multiple branches where some are stable, it's more likely to be reported and fixed in private, but with the development branch either fixed on the same day as a 'bonus' or lagging behind by a few hours or days. This overall doesn't feel too bad, we would want to encourage module maintainers to add a new alpha/beta/rc promptly at least.

    For modules not opted in at all, or which have never had a stable release, then we're looking at more likelihood of issues being disclosed pubicly and possibly not fixed in a timely manner too. The more sites install Drupal CMS, the higher the usage of those modules will be, so this will gradually become more of a problem.

    From a non-security perspective there's also the issue that an alpha release of a module makes zero promises about upgrade path, but we want sites that install Drupal CMS to be able to update when newer releases of modules come out, so included modules really need to be at beta or rc stability or higher from that perspective.

  • πŸ‡¦πŸ‡ΊAustralia pameeela

    We are likely to have some alphas in the initial release, but pretty much only because of D11 support. Project browser is also alpha currently, not sure what point it will be at in Jan though.

    I don't think we can say definitively that we won't ship with any alpha modules. We'd love not to, but it was a battle to even get D11 fixes committed and tagged at all.

    Certainly, all of the modules we'd conceivably be shipping as alpha already have a stable release so I think the risk is heavily mitigated as @catch pointed out. The alternative was to stay on D10 and have our pick of stable branches!

  • πŸ‡ΈπŸ‡°Slovakia poker10

    For modules with multiple branches where some are stable, it's more likely to be reported and fixed in private,

    Unless the issue affects only the non-stable branch - in such case, it can be disclosed publicly (and even if not, the fix will be without SA, so sites will not be notified properly). Not each module has an active development, releasing major/minor versions regularly. Some modules are stuck on a non-stable branch for a long time and differences between branches can be pretty big. I am not talking only about modules included in the Drupal CMS dev branch right now, because this is not only a question about the 1.0 release.

    Certainly, all of the modules we'd conceivably be shipping as alpha already have a stable release so I think the risk is heavily mitigated as @catch pointed out.

    There are some without any stable releases (therefore not covered at all):

    • dashboard 2.0.0-alpha1 (not covered, no stable releases)
    • friendly_captcha_challenge 0.9.18 (not covered)
    • project_browser 2.0.0-alpha5 (unstable, no stable releases)
    • gin 8.x-3.0-rc14 (rc, but still unstable with no stable releases)
    • gin_toolbar 8.x-1.0-rc6 (rc, but still unstable with no stable releases)

    I have not checked all modules

    Personally I do not think that Drupal CMS should beg maintainers to create a stable releases - it should be the other way - there should be an effort from the maintainers, to have a module in a best possible condition, to be included in the stable Drupal CMS (I know this a very idealistic perspective).

    Just one additional example - the Dashboard module is not covered by SA policy, has no stable releases. It has a text "Use at your own risk!" on the module's page. The dashboard recipe is included in the starter recipe, so we are going to install this module to all sites.

    @pameeela do you think this is correct from the security and stability perspective? Should we require at least the modules which are in the starter recipe + in all other recipes, which are installed automatically, to have stable releases? Otherwise I cannot fully imagine what will the Drupal CMS 1.0 (stable) mean.

  • πŸ‡¦πŸ‡ΊAustralia pameeela

    Dashboard, Gin (+ Gin Toolbar) and Project Browser are all essentially Drupal CMS initiatives and their maintainers are working closely with us. Ideally, they all will have a stable release by January but I don't feel that it is essential that they do.

    Right now, it is true that Dashboard is not really ready for use. We hope to have it ready for our release, not sure what more I can say. Whether it will be removed if it isn't stable is probably a call for Dries to make as the project lead.

    friendly_captcha_challenge is basically a library, so that we can ship with the dependencies for friendlycaptcha, because there is no good way to manage front end dependencies in Drupal. But also it is maintained by one of our track leads, and he can tag a stable release when needed.

    Personally I do not think that Drupal CMS should beg maintainers to create a stable releases - it should be the other way - there should be an effort from the maintainers, to have a module in a best possible condition, to be included in the stable Drupal CMS (I know this a very idealistic perspective).

    I think this is a great aspiration, and hopefull will be the case in the future. But the timings of all of this were not ideal -- especially because D10 has LTS and many module maintainers were not in a hurry to get D11 ready -- and it's what ended up happening. As I said, the alternative was to ship on D10 and that was determined to be a worse option. It was also the case that the Drupal CMS team and contributors were able to help many of the modules achieve D11 readiness, so most certainly we have accelerated the pace of D11 adoption which is a great outcome.

    As to documenting policies, if we set strict rules at the beginning before we understand the reality that we are working in, I think it will only create more problems. So we are probably more in a situation where we will document what we have done and create policies for the future, rather than saying now exactly what we will do in January. I realise this is not ideal but it's what happens when you move fast :)

  • πŸ‡¬πŸ‡§United Kingdom catch

    I think this is a great aspiration, and hopefully will be the case in the future. But the timings of all of this were not ideal -- especially because D10 has LTS and many module maintainers were not in a hurry to get D11 ready -- and it's what ended up happening.

    Another alternative would have been to commit to a timeline for a beta/release candidate but not a stable release, to give these modules an extra few weeks to become stable.

    The main concern for me with something like navigation is less the security support aspect (although there are scenarios where that could be very bad), but whether there will be some kind of schema change without an upgrade path while it's still beta, breaking sites that have installed it.

    While Drupal CMS isn't responsible for providing an upgrade path, it is responsible for installing sites which might not have one, so then should only include ones that will. Possibly there are no schema changes for navigation planned or an upgrade path will be provided if there are, but I don't know the answer to that, and nor will anyone else without doing a fair bit of research or talking to the maintainers.

  • πŸ‡¬πŸ‡§United Kingdom catch
  • πŸ‡¬πŸ‡§United Kingdom catch

    @pameela

    If you were starting a new project today that had to use webform, would you really use D10 so that you had security coverage?

    I would use the alpha, but there are mitigating factors:

    1. I've got 20 years of Drupal experience and know how to apply patches and debug things if they go wrong.
    2. If I start a project now and use an alpha release, there is a non-zero chance the project will have a release candidate or stable release by the time that project launches.
    3. I would rather spend time helping get modules stable on a new major release, than deal with a major core update 6 months later. Not necessarily because it's less work, but because it feels like a choice between spending time usefully on things that benefit all Drupal websites rather than busywork within an individual project.

    For the same reasons, I have existing Drupal 10 sites that I probably won't start updating to 11.x until mid-2025 in the hope that the total amount of hours/budget spent on updates will be lower in mid-2025 than it would be now (or in 2026 when Drupal 10 is nearly out of support and contrib has moved on). e.g. if they have 50 modules installed and 30 of them are alpha now but will be stable in May 2025, that's a lot less to worry about.

    However Drupal CMS is primarily aimed at non-technical Drupal users, and my understanding of it is it's specifically supposed to remove a lot of those considerations when building a site.

    @poker10

    But Drupal CMS/Forms recipe/PB will make the decision (without acknowledgment) instead of users, which is the difference.

    Yes this is the difference for me too.

    The webform recipe isn't enabled by default, so it would be entirely possible to remove it from Drupal CMS today.

    If project browser can install recipes, then individual sites can make the decision to install the recipe, or webform directly, but if Drupal CMS's minimum-stability is stable, then project browser should prevent that from happening.

    But then site owners have workarounds they can make (like lowering minimum stability themselves), or as soon as there's a stable tagged release, it would work.

  • Status changed to Fixed about 1 month ago
  • πŸ‡¦πŸ‡ΊAustralia pameeela

    I think this was worked out between this and other discussions. I'll close so any new conversation is based on updated info.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024