Problem/Motivation
If users modify scaffold files (e.g. ".htaccess"), then they might not get changes to this file made in newer versions of Drupal.
Scaffold files should not be modified in Composer-managed Drupal sites; instead, users are expected to use "append" or "patch" capabilities to add their custom content to the Drupal-provided files. Some users do not know this yet, though, so some additional behavior to detect and warn users who modify the generate files is being proposed.
We ask people to modify their .htaccess file in the following use cases, and there is probably an expectation that their modifications will not be lost.
- using Drupal in a subdirectory
- suggesting a rewrite from or to a sub-domain
Additionally hacking .htaccess can help when redirecting to a front-end application not managed by Drupal.
However allowing composer to update .htaccess, web.config or other files is important because those files may change in new versions of Drupal due to security vulnerabilities, new features and so on.
There are three available ways to manage this in my repository: excluding the files from scaffolding altogether, or adding an append configuration (which is ++ for robots.txt, but not applicable to they types of changes in .htaccess), or by patching the scaffold files with cweagans/composer-patches. Managing scaffold file modifications through the scaffold process (i.e. with append or cweagans/composer-patches) is the recommended method, but poses some risk to developers who are not yet familiar with these techniques, and who might be uncertain about how to configure their file-mappings in order to get the desired behaviors. During this learning period, developers are in danger of accidentally losing any ad-hoc modifications that they may make to their scaffold files, as scaffold files are overwritten every time composer install runs.
Proposed resolution
User Expectations
- I would expect to get the latest version of all the various files at the time I install Drupal
- I would expect that if I make modifications to those files, that they’re left alone
- The exception to that would be if there are updates “upstream” to the file(s) [for example, a change to resolve a security issue, which has definitely happened with .htaccess in the past]
- Then, I want some means of knowing there's a change between my local files and upstream, so I can back my stuff up to compare/constrast the two and figure out what I need to do (either accept the changes wholesale or merge them into my modifications somehow)
The scaffold plugin now takes the following actions:
- When a scaffold file is written, its contents are hashed, and the hash values are stored in a cache.
- When a scaffold file is to be rewritten, then the contents of the source file are once again hashed; if the hash value matches what is in the cache (the source file has not changed), and the scaffold file still exists at its destination path, then it is skipped (not written).
- If a scaffold file is modified at the destination path, this is detected because its content hash no longer matches the cached hash value.
If the scaffold file does not exist at the destination path, it does not count as a modified file.
- When a modified scaffold file is rewritten, then the scaffold plugin will warn the user that they may loose data, and will prompt them for the action to take
- They may choose to discard their modifications.
- They may abort the scaffold operation, in which case none of the scaffold files are changed.
- They may permanently keep their modifications, in which case the file is removed from consideration and is no longer managed by the scaffold plugin.
Note that in the last option, where a scaffold file is no longer managed by the scaffold plugin, the user presently (in the current implementation of the most recent version of the patch) will not be warned if the scaffold file changes in a newer version of Drupal. It is desirable if the user knows when a modified upstream file has changed. The proposal is to print a warning during scaffolding. It would not be consistent to prompt, as the user has explicitly removed the file from being managed by the scaffold code. A prompt may be missed in the output of composer install / composer update; however, the user currently is not warned at all e.g. when running drush pm-update, so perhaps this is sufficient.
The hash cache data used by the scaffold plugin must be stored in either a transient location or a non-transient location. The non-transient location was selected.
Discussion on transient vs non-transient hash cache storage
An example of a transient location is inside the vendor directory, e.g. vendor/drupal/scaffolded.json. The advantage of the transient location is that the file does not have to be managed by the Drupal site. It may be committed to the repository if vendor is committed, but users who do this are accustomed to ignoring changes to vendor contents during updates. The disadvantage of the transient location is that it may be inadvertently removed; for example, if the user removes their entire vendor directory and then runs composer install, then the scaffold operation will overwrite any modified scaffold file. This may be unexpected, as scaffold files are written to locations outside of the vendor directory.
An example of a non-transient location is inside the project root, e.g. .scaffolded.json. The advantage of the non-transient location is that it is not subject to accidental removal if the user should decide to delete their vendor directory. While it is still possible for the user to deliberately delete their hash cache, with all of the consequences described above, this is less likely to happen. The disadvantage of the non-transient location is that there would be a new dot file at the project root of every Drupal site, and its contents will change inscrutably during Drupal upgrades. Every site owner must be aware of this file and decide whether to commit or .gitignore it -- although we can place it in example.gitignore to help with this decision, and document the file.
The proposal is to use a non-transient location for now, and add a comment to the file pointing the user to the scaffolding documentation. Since the hash cache keeps transient information that is only useful to users who are not using the recommended scaffold file management techniques (append or patch), we would then have the option to perhaps move the cache to a transient location in Drupal 9 or Drupal 10, once we felt that the recommended method was sufficiently socialized.
Original proposed resolution from becw
- Update the default behavior in core to specify
overwrite: false for some scaffolding files (likely .htaccess, robots.txt, and web.config)
- Update comments in those files to include a note like "If you modify this, update your composer.json to contain ___", since each of these contain comments suggesting how to modify different parts of their contents
- Reduce the frequency that the scaffolder touches any files, so that they're only updated when core is installed or updated, which makes it much easier for developers to manage the diffs -- and allows developers to routinely incorporate core updates that affect these files.
Remaining tasks
- We need to decide what to do about scaffold files that the user has removed from consideration (set to "false" in file-mappings). Currently, no warning is provided if the source scaffold file changes once this is done.
User interface changes
Possible changes to how running composer behaves
API changes
None
Data model changes
Introduces a "transient-cache" field that allows users to decide whether to keep their hash cache in a transient location (inside the vendor directory), or whether it should be stored in the project root, where it may be committed to the repository (the default).
Release notes snippet
TBD