[meta] D8 Extension System: Discovery/Listing/Info

Current action plan

• #2185559: Extension System, Part I: SystemListing clean-up + performance; remove SystemListingInfo + drupal_system_listing() →
• #2188661: Extension System, Part II: ExtensionDiscovery →
• #2208429: Extension System, Part III: ExtensionList, ModuleExtensionList and ProfileExtensionList →
• #2659940: Extension System, Part III: ThemeExtensionList and ThemeEngineExtensionList →
• 📌 Extension System, Part IV: Properly register all installed extensions/namespaces during container generation Needs work
• Extension System, Part V: ExtensionInfo (is this still needed?)
• #3023133: [PP-1] Extension System, Part V: Decouple profiles from modules →

Objective

1. Work on some installer related issues revealed that HEAD contains plenty of insane call chains in code relating to discovering and listing extensions.
2. Some of these call chains are causing an infinite recursion when calling the wrong function at the wrong point in time, because calls are hopping through legacy procedural global state and OO code and back.
3. On top, ~40% of DrupalKernel consists of very complex code and undocumented logic that only exists in order to cope with the fragile legacy extension discovery/listing code.
4. We need to fix this before release.

Goal

• Replace the entire legacy code with dedicated Extension component classes.

Preface: Drupal's Extension Architecture (current)

1. There are 4 extension types: profile, module, theme, theme_engine.
2. Every extension type has a standard file extension; e.g.: bartik.info.yml
3. Extensions are selectively discovered by extension type; e.g.: 'theme'
   1. Extensions of a certain type are expected to be located in a type-specific subdirectory; e.g.: ./themes
   2. Extensions are supplied by (1) Drupal core (2) An installation profile (3) The developer/site builder/end-user.
   3. To discover extensions, the installation profile and its directory must be known; e.g.: /core/profiles/standard
   4. Extensions are searched in multiple filepaths: (1) /core/$type (2) /$profile/$type (3) /$type (4) /sites/$site/$type
   5. If the same extension is found in multiple locations, then the later location overrides the former — unless the later one is not compatible (whereas "compatible" is limited to a Drupal core major version compatibility check as of now).
   6. The list of available/discoverable extensions is not expected to change within a single request at regular runtime.
   7. Extensions are normally not discovered while serving regular page requests.
4. A list of enabled/installed extensions is stored in configuration instead.
   1. The current extension list and their filesystem locations are compiled into the service container.
   2. If the configured list of extensions is changed, the service container needs to be rebuilt.
5. Each extension is required to provide an $extension.info.yml file, which contains meta information to describe the extension (package).
   1. The meta info of an extension/package is normally not used while serving regular page requests.
   2. Info Exception #1: Themes require their full meta information in order to operate. Therefore, the full theme meta info is persistently cached and retrieved from cache on every request.
   3. Info Exception #2: Various administrative user interfaces need to output the human-readable label of a module and thus need to retrieve meta info.
   4. Info Exception #3: Various subsystems/services need to know the installed version of an extension.
      (Applies to contributed extensions only. Core has a fixed version constant defined in code. Custom extensions do not have a version, unless a git commit hash can be determined from .git.)

Inventory/Stock-Check

Data sources

Application environment conditions

Exactly two:

Previously, there was a 3rd condition of update.php, which was not able to assume that the installation profile of a previous major version of Drupal core still exists, but that case is obsolete and gone with Migrate in core.

Proposed solution

Clean Inventory

1. Extension\Discovery
2. Extension\List
3. Extension\Info

Data sources

1. -$settings['install_profile'] = 'standard';
   +$settings['install_profile']['standard'] = 'core/profiles/standard';
   

   The filesystem location of the installation profile is required in order to efficiently discover + rebuild extension lists.

   This setting is not supposed to be touched by site builders/developers either way, so the exact data type/value in settings.php should not matter.

   Additionally, this opens the door to add support for #1356276: Allow profiles to define a base/parent profile → , potentially even post-8.0.0.

2. -$conf['simpletest.settings']['parent_profile'] = 'openatrium';
   +$settings['simpletest_parent_install_profile']['openatrium'] = 'profiles/openatrium';
   

   ...or completely obsolete with (1), because with that, reality is just this:

   $settings['install_profile']['openatrium'] = 'profiles/openatrium';
   $settings['install_profile']['standard'] = 'core/profiles/standard';
   

3. -system.module:enabled
   -system.theme:enabled
   +core.extension:modules
   +core.extension:themes
   

   → /core/config/core.extension.yml (← "core.extension" == Drupal\Core\Extension)

   Introducing core configuration. (restricted to critical base system components)

   Resolving a major circular dependency problem in the installer, tests, and elsewhere: System module cannot be installed through regular means, because System module itself supplies the system.module configuration file.

   Likewise, #2184387: Remove SchemaStorage → discovered that System module defines some basic configuration schema data types that have to be available at all times. But yet, for concistency, DX, discovery, and code re-use purposes, it would be preferable to keep defining them in a YAML file like all other config schema files.

   → /core/config/schema/core.data_types.yml

   The idea is to turn "core" into extension and extension type itself.

   So as to allow it to be a regular data provider like any other extension type. The only exception is that there is only one core, so $extensionList->getPath('core', 'whatever') will always return "core" (the extension name is ignored).

   Starting from core services (core.services.yml), to core libraries (core.libraries.yml), to base config schema, and base system extension (default) configuration, core already is a data provider like any other extension:

   /core/config/schema
   /core/config
   /core/lib
   /core/tests
   /core/core.libraries.yml
   /core/core.services.yml
   

Blockers

The primary purpose of this meta issue is to (1) raise awareness of the (non-obvious) problem space, (2) discuss the concrete proposal presented here, and (3) come to an agreement ASAP.

Given that other issues required me to make myself familiar with all of this code/insanity, I'd be happy to volunteer on my own to code up these gems, but before doing so, I'd like to achieve at least some basic level of agreement. (The devil is in the details anyway.)