Proposal to manage the Drupal core release cycle

TL;DR

• There are several issues with core development as it stands: not innovating fast enough, insufficient commercial incentive to contribute, contributor burnout, and unpredictable release cycles.
• We need to think about how to solve these not only for core developers, but also contrib developers, technical site owners, and non-technical site owners. They each have a different set of needs and challenges.
• In addition to starting from Larry's core conversation on the Drupal release cycle, we evaluated other projects' approach to release management, and attempted to create a framework that could work for our community, which entails:
  • Release new minor versions on a timed schedule: every 6 months, until the final minor release, release a new minor version that includes mostly-non-BC-breaking improvements and additions.
  • Only support the latest minor release of any given major version.
  • Have a single LTS release per major version, which is the final minor release of a major version.
  • Provide both security and regular bug fixes to this LTS release until 3 months after the next major version's (e.g., 9) LTS release is available, and continue providing security fixes until 3 months after the following major version's (e.g., 10) LTS release is available. This adds more work for the security team (e.g., Drupal 6 would be security supported until 8's LTS rather than only until 8.0), so we'll need to figure out how to make it possible.
  • Only branch the next major version of Drupal when there's something big enough completed to justify it, and these will probably continue to release every 2-3 years.

Introduction

In #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x → , there has been a lot of talk about Drupal 8 adopting semantic versioning, where version numbers and the way they change convey meaning about the underlying code and what has been modified from one version to the next. While we do certain classes of changes and additions during stable versions now, there's a lot of things we don't touch until we do major versions, in part due to limitations of our current 2-digit version scheme, and in part due to a culture of shifting focus to the next major version right away.

I believe semver is a great framework to adopt. The tricky part is in the details. I propose we look at this problem through the eyes of four personas.

Personas for Drupal core

Persona 1: Drupal core contributors. They'd like to see more frequent releases to increase the velocity of innovation happening in core, and to reduce the time commitment required to get innovations into the next release. Right now, we couple innovation in our admin UIs, markup, APIs, and internal implementations of subsystems to huge releases that re-write half of core. Not only does this result in a multi-year cycle to get new functionality released, but it also makes development of even minor functionality and UX improvements difficult because it requires constant patch rerolls as things shift.

Persona 2: Drupal contrib maintainers. Contrib authors may come across core limitations that either block their porting or development progress altogether, or require sub-optimal workarounds that are difficult to maintain. The faster these get resolved, the better the entire Drupal ecosystem is for it users. On the other hand, contrib maintainers also require a stable core API to work against so that once they release a stable version of their project, they don't need to worry about it breaking with every minor core update. This means we need to be very careful in what we do and do not break between minor releases.

Persona 3: Users without ongoing development resources. A lot of people get their Drupal site up and running and pretty much stop spending time or money on it, either because they don't have the resources or because they don't want to. Other than dealing with the occasional security updates, they don't want to be bothered upgrading themes or modules. Ideally, Drupal would expand the Update Manager feature to automatically upgrade Drupal core from one minor or patch version to another, to allow for both innovation and security releases to be less disruptive to this audience.

Persona 4: Users with ongoing development resources. Some users have maintenance contracts, internal staff, or sufficient free time to improve their site over time, to upgrade Drupal modules, etc. These people are willing to both chase and contribute to the latest functionality if it benefits them, but don't want to do so at the cost of a major, multi-thousand dollar upgrade.

How to balance these audiences' needs?

Let's look at some of the details of the Drupal core release cycle if we adopt semver:

1. How strictly do we preserve backward compatibility (BC) between minor releases?
2. How often do we release new minor versions?
3. How long do we support the previous minor version once a new one is released: i.e., do we provide an overlap window to give people time to update?
4. Do we mark one or more releases as a "long term support" (LTS) versions of Drupal? If so, when and for how long?
5. How will contrib authors and security team members be able to cope with more frequent releases of core?
6. What happens with major versions?

With these four personas in mind, I think the best way to go is with:

1. Preserve BC fairly strictly. Not 100%, but close. We should only allow for trivial BC breaks (e.g. breaks that affect a very small number of contributed modules or themes), to minimize impact on the personas that cannot easily fend for themselves. This means that the three main approaches to innovation are: (i) providing a backward compatibility layer (e.g., if moving a particular hook to an event, then continuing to fire the hook as well), (ii) providing a separate, alternative implementation (e.g. core could ship with an experimental field UI as a separate module) and (iii) adding completely new systems (e.g. a new entity or field type could be moved into core).
2. For predictability, release a new minor every 6 months. To start with, I suggest 3 months of code thaw, followed by a 6 week beta period followed by a 6 week RC period. With time, I would like to lengthen code thaw and shorten beta and RC if we find that we are able to do so. Note that even during the code thaw period, every patch committed to an already released major version's branch must not knowingly introduce any release blocking follow ups. It's okay to introduce nice-to-have follow ups that if completed would make things better, but if not completed, the next minor release can still happen on schedule.
3. No support overlap window for minor releases. Once a new minor is released, drop support for the old one. So long as only trivial BC breaks are allowed and we provide guidelines for people on how to mitigate them, there's no reason for anyone to stay on an old minor. Site owners unwilling to keep up with minor updates should wait for the LTS release. However, the initial (.0) patch release of a new minor version must not itself contain security fixes that haven't been backported and released to the prior minor; this might in some cases require releasing a final patch release for the prior minor on the same day as releasing the new minor (e.g., release an 8.0.6 on the same day as 8.1.0, but from that point on, no longer support 8.0).
4. Make the last minor release, and only that one, the "Long Term Support" (LTS) release, fully supported until the next major version's LTS plus an additional 3 months to allow for migration time, and security supported until the LTS after that plus 3 months. In other words, site owners can migrate from 8 LTS to 9 LTS and continue receiving all bug fixes, or from 8 LTS to 10 LTS, but during the last couple years that they're on 8, only receive security fixes.
5. To help ease contrib porting, we could hard freeze the code of a new release (only allow commits that fix release blockers) during the RC period of 8.X.0 (and clearly communicate this) to provide contrib a window to update their code if necessary prior to site owners updating to the new release.
6. Approximately 1 major version every 2-3 years. For major versions, we would hold off opening the next major branch for committing patches to until that is needed. This might mean, for example, that 9.x isn't opened until after 8.2 or 8.3 is released (i.e., 12 to 18 months after 8.0 is released). This enables patches to Drupal 8 to be committed to Drupal 8 directly, without needing to first be committed to Drupal 9 and then backported to Drupal 8. However, this means that people working on major rearchitecture of Drupal during this time (e.g., 🌱 Evaluate Symfony form component as a replacement for Drupal FAPI Postponed: needs info and #1843798: [meta] Refactor Render API to be OO → ) must do so in sandboxes. Integrating multiple such initiatives or proceeding with mass conversions to the new architecture requires 9.x to be opened, so we'll open it when such initiatives are far enough along to warrant that. Once we open that branch, we would also freeze the currently stable major and begin preparing its final (and therefore, LTS) minor release. Therefore, in order to not have too long a time go by with no innovations in core, we would also target releasing the next major version within a year or less of opening its branch.

What is the practical application to these audiences?

Given these set of recommendations, and nearly complete backward compatibility, here is what it means for each of the personas.

Drupal core contributors: Core developers can improve core and use those improvements on real sites faster, so long as they (mostly) preserve BC. More frequent, less backwards-compatibility-breaking releases of Drupal core would allow us to move culturally to a more agile development approach of getting smaller changes out faster, seeing how they do, and making further changes based on real-world data. This move also potentially allows us to build core more sustainably, by giving core developers the option to tie some of their development to billable client work, which can only happen if meaningful changes are released more often than once every three years.

Drupal contrib maintainers:
Contrib maintainers can expect their modules and themes to keep working with new minor releases of core, most times with no changes needed at all, sometimes with new integration code needed for whatever new implementations are added to core, and very rarely with some small fixes needed to accommodate a trivial BC break. There will be at least one release candidate for each minor release to allow contrib authors to test their code in advance of site builders updating their sites.

Users without ongoing development resources: We will need to provide guidelines to these users, such as either stick to LTS releases or:

• Don't use custom modules; stick to only contrib modules that are adequately maintained. Or, if your site requires custom code, make sure the developers you hire know to limit their code to only accessing APIs marked as stable (for example, if $form arrays were marked as unstable APIs, don't use hook_form_alter()).
• Since markup of admin pages might change in minor ways, stick to core/contrib admin themes; don't use a custom theme.
• Other guidelines TBD.

So long as these site owners stick to these guidelines, they should be able to update to new minors as easily as updating to new patch releases.

Users with ongoing development resources: These site owners can use custom modules and themes, as well as contrib projects that they help maintain, to the extent that they're able to keep them up to date with minor core updates. Since only trivial BC breaks will be allowed, this will not require too much of their resources. More of their resources would be needed if they want to use and integrate with whatever new features/implementations are available in the new release, but they can decide which of those to use and when.

How does this approach compare with other projects?

We compared and contrasted release strategies with a number of different projects: WordPress, TYPO3, Joomla!, Firefox, Ubuntu, PHP, and Symfony. All have a substantive release anywhere from every 6 weeks (for Firefox) to 1 year (for PHP), with most at around 6 months. The projects differ in whether or how they provide LTS releases: Ubuntu does so once every two years and supports them for 5 years, while WordPress only provides security support for their latest release, but tries hard to always preserve BC, even in major releases.

The proposed approach attempts to take the best ideas from these projects, and put them into a framework that can work for our community. The approach of mostly BC-preserving but innovative releases every 6 months maps closely with most of the projects studied, providing a single LTS release as the final minor maps closely with Joomla!, and supporting an LTS release until after two additional ones are released maps closely with Ubuntu.

Next Steps

• Gather feedback on this proposal. If generally positive:
  • Evaluate the current backport policy and determine how it should change for 8.x+ (separate issue).
  • Develop a plan for what infrastructure changes are needed to the Drupal.org packaging system, update status module, Drush, etc. to allow for this move (separate issue).
• Else, iterate and come up with an amended proposal, then do the above. :-)

I think it's important for us to hash this out now that Drupal 8 has positioned us via its new architecture to support iterative innovation, and hopefully grow the core development team as well.

Questions? Comments? Please provide feedback about this proposal.

Big thank you to catch, alexpott, webchick, Crell, effulgentsia and xjm for helping me create this proposal, and to all the participants on #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x → , whose comments have also influenced this proposal.