[meta] Composer Initiative Phase 1: Add composer build support to core

This is the main Drupal core issue for the Composer initiative: https://www.drupal.org/about/strategic-initiatives/composer → (From this proposal: 🌱 Proposal: Composer Support in Core initiative Fixed )

Note that the Composer initiative has its own d.o project, but this is mainly a place to store meeting notes: https://www.drupal.org/project/composer_initiative →

Problem/Motivation

Make an official method for starting a Drupal site with composer.

Current releases of Drupal do not have an easy way to make use of Composer-based dependencies. Some contrib modules require that Composer be used to install them.

Proposed resolution

Add a composer kickstart template, along with supporting packages and change the build process to utilize them.

This means that the 'tarball' version of core that the user downloads from Drupal.org will have been built using Composer, and will not break when they need to use Composer to manage it.

Roadmap

1. We need a way to handle the 'scaffolding' of files so we can build a product with composer:
   #2982684: Add a composer scaffolding plugin to core →
2. We need to reconfigure drupal core to leverage that scaffold plugin.
   #3067645: Add core scaffold assets to drupal/core's composer.json extra field →
1. But first we must remove the wikimedia merge plugin because it interferes with our plan, and causes many other problematic issues
   #2912387: Stop using wikimedia/composer-merge-plugin →
3. Once core is setup such that a template can take advantage of it, we'll need to create the template files that packaging can use to generate tarballs that mimic the existing tarballs drupal.org provides.
   #2982680: Add composer-ready project templates to Drupal core →
1. In order to do that, however, we'll need to be able to have a known set of locked dependencies that core 'ships' with, and rely on that
   #3076600: Create drupal/core-recommended metapackage →
2. And We will also need to make sure that all of the existing safeguards are in place and our 'webroot drupal' doesn't ship with test files.
   #3057094: Add Composer vendor/ hardening plugin to core →
4. Once all of that is in place, we'll modify drupal.org's packaging to build the tarball from the template instead of the git clone.
   #3071726: Package tarballs using composer template if available →
5. Once the above is complete, we'll have a solid milestone that will mean all future drupal installations will be able to start with a composer ready tarball or be able to use an officially sanctioned way to start a drupal project.
6. We'll need to ensure we've got good tools for converting existing sites to get everybody on the same path moving forward:
   #3047468: Create a tool to convert a non-Composer-based Drupal to a Composer-based one →
7. Various minor cleanups/bug fixes/DX/UX improvements
1. #3084298: Make composer installer paths consistent between drupal/drupal and the project templates →
2. #3084326: Install .editorconfig and .gitattributes at the project root →
3. #3082866: composer complains "Skipped installation of bin bin/composer for package composer/composer: file not found in package" →
4. #3085075: Add '-s dev' to Composer Project Template documentation so instructions work w/out a stable release →
5. ✨ Add gitignore(s) to Composer-ready project templates Needs work
6. #3082983: Decide what to "suggest" in Composer Project Templates →
7. ✨ Prompt users when adding new requirements with scaffolded assets Needs work

Documentation Updates

We will need to inform the users about all of these features

1. Update the user guide and documentation guides for sitebuilders
   https://www.drupal.org/project/user_guide/issues/3086656 →
2. Update documentation for developers
   [ ISSUE PLACEHOLDER ]
3. Create a change record that discusses the differences between the dev repository and the built product(?)
   [ CHANGERECORD PLACEHOLDER ]
4. This issue probably has more nooks and crannies we can look for composer doc info.
   📌 Create a list of Composer topics that need documentation Closed: outdated
5. Another request for docs/instructions:
   ✨ Add composer-based install instructions to the Drupal project page Closed: outdated
6. #3086655: Update INSTALL.txt to account for composer installations. →
7. #3000374: Improve Composer update workflow in UPDATE.txt →

Testing Updates

In order to prove that a lot of this stuff works, we need to add a new test type to the infra for build and upgrade types of tests.

1. So we'll need to add the new testing type
   #3031379: Add a new test type to do real update testing →
2. Then we'll need to create tests
   #2984031: Create Build Tests For Composer and Drupal →
3. Finally, we'll need to have the testing infrastructure execute our tests
   [ DRUPALCI ISSUE PLACEHOLDER ]

User interface changes

None

API changes

None

Data model changes

None

Build changes

None