Prevent modules which have unmet Composer dependencies from being installed

Problem/Motivation

In Drupal 8, contributed modules may require Composer dependencies, but Drupal core itself will not require site builders and developers to use Composer to assemble a site's code base. This means that contributed modules currently have to ensure their Composer dependencies are installed themselves, which is a task too complex and important to reinvent in every project.

In order to solve this problem we need to determine two things:

1. Does an extension require composer to be used to install its dependencies?
2. Has composer been used to install its dependencies?

1. Does the extension require composer

We propose adding a build_dependency[] key to info.yml files that allow extension developers to explicitly declare that composer is a build requirement.

This requires extension maintainers to take an extra step of declaring composer as a dependency.

It has some added benefits where that metadata could be useful beyond just Drupal's runtime/install time. For instance, we could modify project pages to alert users without composer abilities that they cannot use the extension in the first place

2. Has composer been executed for the extension

If we determine that an extension requires composer, we must then determine whether or not that requirement has been fullfilled. If it has not, we need to alert the user that a module cannot be installed, or needs to be uninstalled (Already installed modules may go from not requiring composer to requiring composer).

There are couple of proposed approaches to this as well:

• A. Validate that the dependencies that are declared are fullfilled.
• B. Inspect a build artifact that lists the installed extensions to see if an extension is listed, and if it is not listed, assume that it still needs to be executed

Solution A would actually calculate the required dependencies. Since composer supports a variety of dependency declarations (replace, provides, requires, conflict etc), the only sure way to calculate would be to use composer code itself to calculate. Composer's dependency calculation is *extremly slow*, and cannot likely work in an environment where there are web requests involved. Because Composer is a build tool, and should not be utilized at runtime for codebase validation.

Solution B gives us a list of extensions that have been installed with composer. There are two proposed implementations for this solution:

• a. Inspect vendor/composer/installed.json to determine if an extension has been installed with composer
• b. Run a composer build script that creates core/composer/installed-extensions.json that gives us a list of what has been installed via composer

Implementation a. uses an artifact from composers build process, but is not part of a published api, and as such might someday change.
Implementation b. shields us from those changes, and gives us complete control over the format of the file we're using to check.

------------------------------

Stating the problem:

These are composer dependency declarations supported by packages.drupal.org (from @Mixologic #309)

• A. simple, composer.json in the base project dir. (project/composer.json, project/project.info.yml)
• B. "Module at the root level of the project contains all of its submodule composer package dependencies" (project/composer.json, project/commerce.info.yml, project/commerce_order/commerce_order.info.yml) - where commerce_order's package dependencies are declared at the root of the *project*, not in commerce_order's directory.
• C. "Parent module at the root level contains composer dependencies, and a submodule contains *different* package dependencies, like the inmail example above.
• D. "module at the root level of the project has no composer.json, but a submodule does: http://cgit.drupalcode.org/cloud/tree/?h=8.x-1.x && http://cgit.drupalcode.org/cloud/tree/modules/cloud_service_providers/aw...
• E. "No parent project at the root level, only submodules, each with their own dependencies" (something like http://cgit.drupalcode.org/cod_support/tree/, where the project is really a bunch of related modules, but there is no "parent" module.
• F. Sometimes there is *more than one module* in the same directory. I dont know if this can still happen in d8 (http://cgit.drupalcode.org/stringoverrides/tree/) and hopefully it doesnt, cause ew.

Situation #1: Non-Composer users will install Drupal 8 from a tarball. They will place contrib modules in modules/ using tarballs.

Desired behavior: If a contrib module has a Composer-based dependency which is not present, the user will be prevented from installing that contrib module. All the restrictions of hook_requirements() returning REQUIREMENT_ERROR will be present: The status page will show an error, the site installer won't work, and update.php will not be able to perform an update.

Situation #2: A user is using a contrib module installed as a tarball. This contrib module has an update, and in the update, it introduces a composer.json file which means there are now Composer-based dependencies.

Desired behavior: The same as #1, except the module is already installed so we can't prevent that. We can show the error in the status page, and we can refuse to update. It might be the case that the module will cause a crash, but that's outside of scope here. We only want to make a good-faith effort to tell the user they should start using Composer.

Situation #3: A user is using a contrib module installed as a tarball. This module has Composer-based dependencies which are always present in Drupal 8 (such as a symfony component). An update to the contrib module changes the dependency version constraints to a higher version than is present in Drupal 8.

Desired behavior: Prevent the user from performing the update to this module, because its dependencies are not met.

Situation #4: A user has contrib modules installed through a mixture of tarball installation and Composer-managed installation. The tarball and Composer-based modules have conflicting requirements and should not be able to both be present.

Desired behavior: Prevent the user from updating the tarball module, which should inform them that they should use Composer to manage that module (once all the conflicts are resolved).

Proposed resolution

General rules of thumb for proposed solutions:

1) Do not make Drupal have a dependency on composer/composer.

2) Introduce as few complexities as possible, in order to leave room for more robust solutions within the Composer layer.

Proposal #1: Look at vendor/composer/installed.json

From #261 → :

The situation is: A user is trying to install an extention, where:

• The extension has a composer.json file associated with it
• The extension's composer.json has evidence of external dependencies (has a require on something other than drupal/*)
• The extension name does not show up in installed.json

Then we can assume that, regardless of whether or not that particular extension contains any replace or conflict information, that composer has not been run to retrieve it's dependencies, and that it was installed using some other method than composer.

We do not have to concern ourselves if a module has a require on anything drupal/* because they will be one of the following:

• drupal/core - which will always be satisfied.
• drupal/core-[component] which will always be satisfied.
• drupal/[project/module] which should be specified as a dependency in info.yml. I cannot think of a case where an extension requires a project to be downloaded, but does not require that project in order to install that module.

When we have library types on drupal.org, that are not subtree splits of core itself, we may need to have a namespace for those (drupal/lib-libraryname) so we do not count them as a filled dependency. (could also be drupal-lib/libraryname too. )

Proposal #2: Build our own copy of installed.json

Require all modules that are part of a Composer package to be installed/built through Composer prior to installing/enabling it within Drupal.

This is done by compiling a list of Drupal extensions that are Composer packages, and their installation paths during composer install and composer update. Whenever a module is enabled, we check if it's part of a Composer package, and if the extension's path lies anywhere within the paths of installed packages, Drupal considers it installed/built through Composer. This is based on Composer's own restriction that it cannot install one package into another.

An advantage of this approach is that the coupling between Drupal and Composer remains small, and extensions can take full advantage of any requirements specifications and package links Composer allows, as well as class autoloading without having to bootstrap Drupal.

Remaining tasks

• update the issue summary, listing a (summary) history of the proposed solutions, the pros and cons, and what concerns people had (but may not anymore, or still have) with links to relevant comment numbers. This can help reviewer be up to speed on this issue, and also help ensure that people see their concerns are heard correctly (even if there may not be consensus).
• (maybe) update the issue summary with a list of criteria for this issue (start with what there is consensus on, then list criteria for a solution that people dont yet agree on, the pros and cons and concerns people have for if an item should be required for this issue. The criteria can help people evaluate the proposed resolutions.
• (maybe) look at https://www.drupal.org/governance/core → and list which sign-offs this issue needs (and keep the list in the summary as "done" (with a link to the comment number) or "todo")
• (maybe) update the issue with some links to key comments from reviews, and (maybe) include the role of the reviewer [experience with this issue | maintainer with role X of project Y | drupal.org CI maintainer | ... other]
• (maybe) list some personas for this issue: core contributors, core maintainer of X component/module/system, contrib maintainer, contrib contributor, drupal.org CI maintainer, site builder, site administrator, content creator, site visitor, more? ... and what the impact/risks/benefits/concerns are (and should be?) for each persona.
• update the issue summary with what follow-up issues might be needed

User interface changes

One additional Composer item will appear in the status report, indicating whether all necessary Composer dependencies have been installed or not. This is the same requirement that can prevent modules from being installed.

From #120 Jan 6 2016

API changes

No code compatibility breaks.
Potentially adds a new api for extension developers to express their dependency on a particular build tool.

This requires any module with a Composer file to be part of the Composer build, before it can be installed within Drupal.