Related initiatives
Problem/Motivation
Myself, along with some of my co-workers, were tasked by Dries to perform a series of interviews along with other research to try and determine what the main blockers are for Drupal 8 adoption, as well as what factors are leading to Drupal 8 being hard to use. (These findings will be part of the Driesnote at Nashville.)
We've talked to over 50 people at this point, from site builders, to developers, to support and sales people, to trainers, both enterprise-level and small-medium business level, from various points on the globe including North America, Europe, and India. From this research, we feel we've gleaned a solid set of common pain points and have some suggested next steps.
One consistent pain point that came from all walks of life, from the initial evaluator to the non-technical site builder to the hardcore D7 developer trying to make the jump to D8, to the poor schmuck stuck maintaining a site after the initial developer is gone... Composer. :\ From a Drupal.org perspective, it's still very well-hidden that this is even a thing, until you run smack into a module that requires it, requiring a migration job. The instructions on how to use Composer are unclear and provide already confused people with multiple options on moving forward. Core doesn't embrace Composer best practices OOTB, requiring third-party solutions often held on obscure GitHub repos. And even when you get it working, the errors that come back are obscure and difficult to troubleshoot.
The details of this particular problem are
If a user
- Starts a drupal project with the current drupal tarball OR
- Starts a project with `composer create project drupal/drupal` OR
- Starts a project from a git clone or copy of the git repo
And then attempts to use composer to manage their site, they will invariably be set on a path of misery, confusion, and frustration because:
- If the user adds a module with composer, their composer.json and composer.lock will have changed from what is shipped with core. When they need to upgrade core, following the documented manual upgrade will result in conflicting changes in the composer.json and composer.lock file. We’ve always said, ‘don't hack core’ but using composer in the scenario forces you to do that. And they’ll have to manually resolve conflicts in multiple json files. Not fun by any stretch of the imagination.
- There is no way to use composer to upgrade drupal core in this scenario, so they have to upgrade manually.
- The users who are most likely to get stuck and have problems are the ones who have been doing drupal this way all along, and finally stumble across a module they need that requires that they use composer to install it, and wander into a situation that sets them up for failure. This is because we’re not providing an official, sanctioned, solution.
The root cause underlying many of the issues we face is the result of using Drupal core’s repository as *both* where we do our development work *and* as a ‘product’ that we ship.
Proposed resolution
Conceptually separate Drupal, the product, from Drupal’s git repository, and provide a mechanism that creates a composer ready drupal installation
Phase 1 (scope defined by this initiative)
- Start thinking of the Drupal git repository as a ‘monorepo/manyrepos’ (https://speakerdeck.com/fabpot/a-monorepo-vs-manyrepos?slide=1) situation, where the git repository is where core development takes place.
- Git clones of Drupal core are for development purposes only.
- Directory layout would stay the same, and be capable of behaving as a Drupal site for the purposes of development
- composer.lock file in the root of the git repository would no longer conflict with existing sites, and is to ensure that developers are all working with the same set of dependencies.
- Provide, in core, composer project designed to allow users to “kickstart” a Drupal project.
- Similar to the current drupal-composer/drupal-project template, but even simpler.
- Would exist on packagist.org as a starter project by using the existing subtree split tool
- In order to start a Drupal project, the end user either starts with a
composer create project drupal/kickstart my_project OR they download a tarball that had been constructed using the kickstart
- The kickstart plugin would depend on a scaffolding plugin that is also embedded in the core repo that deploys the scaffold files from core into a sane directory layout for the product.
- An additional embedded package would provide for a list of stable dependencies (similar to https://github.com/webflo/drupal-core-strict)
- Using a new composer namespace allows us to break from a giant chunk of bloated metadata that is the current drupal/drupal and avoids conflicts with accidental attempts to upgrade from a current drupal/drupal site to the new kickstarter.
- Introduce build testing to prove that we can create products out of core’s repository.
- Alter drupal.org's packaging of drupal core to generate our current tarballs using this method so that composer users and non-composer users would still all have the same starting point
- Provide an upgrade path for any existing sites to turn them into a composer ready site, so that the site behaves as if it were built with composer in the first place. This would ideally be a core-native tool so it tests alongside core. Note that there are existing tools to provide this:
Composerize →
and drupal/console, among others.
- Update all documentation to instruct users to *only* start with composer or with the composer created tarball, and provide separate instructions for developing for drupal core.
- Integrate the existing composer documentation into the various locations that are more user focused so that the options for extending a site are under the documentation for adding a new module, and instructions for upgrading a module are solely under upgrading a module, and not under any 'composer specific' documentation on drupal.org. Composer is a tool, not a use case.
There are several additional advantages that we gain by splitting these things conceptually:
- The core git repository can have development resources for drupal core development, and not have to treat those resources as API's for contrib, including their location (phpunit.xml.dist/eslintrc/stylelint.json/etc/etc)
- The plugins and kickstarts would be in the core repo, meaning we wouldnt need to enact duplicate policies and access management for additional repositories that are not drupal core. All of our policies surrounding contribution would stay intact.
- For users with modest needs, that do not need to rely on a module with composer dependencies, this change would be transparent to them and they would still be able to manage a site "the old way", but their site would be 'composer ready' if they did end up needing to switch to managing it with composer.
- When Drupal the 'product' becomes something 'composed' of various components inside of the core repo, we would have the opportunity to offer additional 'kickstarts' that might even allow for minimal versions of drupal, or pre-extended versions, like distributions, or simply variable filestructure layouts supporting both docroot/webroot, and one directory type hosting scenarios. We could have starting points with 'experimental modules' and those without, for example.
- The scaffolding plugin could be reused by distributions and other staring points, reducing maintenance for distro maintainers.
- Some of composer's performance penalties are a result of how we have structured our cascading composer.json's. We may be able to get great performance wins by doing some metadata restructuring.
- Figure out how multisite drupal can be managed with composer - what are the limitations, what's possible etc.
Phase 2/3/....
Once we've stablized this workflow, there are several other things we can continue to incrementally improve:
- Improve the export quality of our components.
- Add composer.json files to all modules in core, and have them be requirements of drupal/core, and offered by the composer facade
- Figure out how to separate php code from assets in modules and in core so that code can be vendored, and assets under the docroot.
- Create an automatic updates feature, make Composer-friendly.
- Make patching a site a default feature of the kickstart plugin (cweagans/composer-patches)
- Support real semantic versioning for core and contributed modules.
- Support tarball generation for Composer managed Drupal distributions
- Contribute upstream to Composer to make it more usable for novice CLI users (more informative output, better troubleshooting documentation) [This could probably move up, and be worked on in parallel, but the point being it's not a blocker for core improving its Composer support.]
- Improve composer performance issues wrt memory and cpu upstream
- potentially offer a composer as a service
- Implement UI on top of Composer / make update manager use composer
Remaining tasks
- Get framework management + release management sign-off on idea <= we are here! :)
- (After that) Develop more detailed implementation plans, including biksehedding the implementation details (in other words, please don't delve into "solution A vs. solution B" details here... ;))
User interface changes
Changes to Drupal.org project pages to feature Composer instructions.
Changes to Drupal.org documentation pages about module and theme installation/updates to include Composer information.
API changes
Alert users to the fact that if they are currently running their site off of a git clone of drupal, that they are on a path to 'not having a good time eventually', and that running production sites from the git repository is unsupported, frowned upon, and contributes to global sadness.
Data model changes
None. The database wants nothing to do with composer.
