Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable.

#73 is working for me with Drupal 9.5.9, however I saw something else that I couldn't find addressed anywhere else that I thought was worth adding here in case anyone else encounters the error I did. I moved a site to a new environment with this in the composer.json:

    "extra": {
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            },
            "file-mapping": {
                "[web-root]/sites/development.services.yml": "false"
            }
        }

...and when I did the initial composer update, scaffolding failed with the error:

In ScaffoldFilePath.php line 135:
                                                                        
  Scaffold file false not found in package drupal/recommended-project.

I removed the file-mapping and and did composer update, and it completed successfully (including scaffolding). I put file-mapping back and subsequent updates worked as well with no errors. So I think it's just a matter of having to exclude this for an initial install.

@tedbow discovered this is also getting in the way of Automatic Updates: #3362143-21: Install rsync on DrupalCI and run build tests with it → .

AFAICT the current issue summary is wrong?

It says:

the only time the failure reported in this issue should be seen is:

1. The site builder modifies sites/default/default.services.yml or sites/default/default.settings.php. (Should be rare, as these files should be copied before they are modified.)
2. The site builder removes sites/default/default.services.yml or sites/default/default.settings.php.

But it's AFAICT also a problem whenever core updates one of these files.

The last 10 times that sites/default/default.settings.php was changed:

$ git log --pretty=format:"%ad%x09%<(25,trunc)%s" -n 10 origin/10.1.x -- sites/default/default.settings.php
Wed Apr 19 11:18:37 2023 -0500  SA-CORE-2023-005 by ben..
Tue Apr 11 14:10:23 2023 +0100  Issue #3027639 by catch..
Sun Mar 12 20:06:51 2023 +0000  Issue #3107548 by tunic..
Thu Feb 23 16:22:19 2023 +0000  Issue #3317265 by ressa..
Thu Feb 16 22:36:17 2023 +0000  Issue #3333281 by Musta..
Wed Nov 30 17:35:34 2022 +0000  Issue #3032746 by mfb, ..
Thu Nov 17 14:13:32 2022 +0000  Issue #3260401 by idebr..
Mon Oct 3 14:17:48 2022 +0100   Issue #3096101 by quiet..
Mon Aug 8 11:39:05 2022 +0300   Issue #3262674 by tstoe..
Wed Jul 20 10:11:30 2022 -0500  SA-CORE-2022-012 by cml..

👆 5 times in the past first 5 months of 2023!

Same for sites/default/default.services.yml:

$ git log --pretty=format:"%ad%x09%<(25,trunc)%s" -n 10 origin/10.1.x -- sites/default/default.services.yml
Mon Mar 6 17:14:57 2023 +0000   Issue #3150614 by pfren..
Fri Mar 3 16:08:14 2023 +0000   Revert "Issue #3150614 ..
Fri Mar 3 11:13:53 2023 +0000   Issue #3150614 by pfren..
Thu Feb 23 10:20:36 2023 +0000  Issue #3198868 by dpi, ..
Sun Oct 9 12:06:21 2022 +0100   Issue #3112452 by lalit..
Mon Oct 3 14:30:44 2022 +0100   Issue #3305748 by kay_v..
Wed Sep 28 11:52:42 2022 -0500  SA-CORE-2022-016 by fab..
Wed Sep 21 14:49:58 2022 +0100  Issue #2381797 by Tom V..
Mon Feb 14 17:23:42 2022 +0000  Issue #3166449 by ravi...
Wed Aug 18 09:53:24 2021 +0100  Issue #2473875 by znero..

👆 4 times in the first 5 months of 2023!

AFAICT that means the disruption is much bigger?

The data in #92 + the fact this has 125 followers makes me believe this deserves a priority bump.

The problem here is the main default folder permission, as in sites/default/default.services.yml.

We need to somehow let Composer deal with the permission of the default folder. For example, if I manually set sites/default to 777, Composer should be able to run and scaffold the files like default.services.yml or default.settings.php.

Please note that this issue summary mentions this in the historical context section:

If you've installed and run Drupal, the sites/default directory receives permission hardening. This can break the scaffolding plugin.

It's not recommended to run Composer as root. But is there a way to let Composer change the permission of the default folder temporarily, without introducing any security holes? Should Composer scaffolding be responsible for the permission hardening and let Drupal check only if the default folder is hardened?

Interesting, @xmacinfo! I like your proposal! 🤓

Maybe a solution could indeed be to let Drupal's Scaffold plugin not apply the scaffolding for non-essential files, such as sites/default/default.* if file system permissions do not allow Composer to overwrite them.

(I say these are non-essential files because they do not affect the live site in any way, they merely serve as starting points.)

OK. I like that as well.

We let Drupal do its own hardening while we modify the Composer scaffold plugin to :

not apply the scaffolding for non-essential files, such as sites/default/default.* if file system permissions do not allow Composer to overwrite them.

.

Should the Composer scaffold plugin display a notice when it skips a file due to insufficient permissions?

Another solution to this problem is to rethink the need for the sites/default/default.* files all together.

For instance for defaults.settings.php is used by the installer to make settings.php and core/Install.txt has manual instructions to use this file to create settings.php. But all this logic was made before we had core/assets/scaffold/files/default.settings.php.

It would be much easier for the installer to just use core/assets/scaffold/files/default.settings.php.
and core/Install.txt to be updated to reference this file.

Then sites/default/default.settings.php could removed entirely and the scaffold would not have the problem of trying to move the file into folder that should be write protected.

This would make Automatic updates much easier as we would not have try to make this folder writable and then re-harden it again which seem dangerous if for some reason we get some kind of hard failure and can't re-harden it.

It seems like it is only for legacy reasons that we have both sites/default/default.settings.php and and core/assets/scaffold/files/default.settings.php and we have to have a test make sure they are identical(i think)

We could even make a new
sites/default/default.settings.txt
that just explains what happened for anybody looking for the file in the old location.

really like the idea #97.
This reminded me of the issue #1672986: Option to have all php files outside of web root. →

Doesn't using scaffold files directly pretty much defeat the point of having scaffold files in the first place, aka the ability to customize them.

Also, I don't really understand how that would solve the problem, the problem with permissions is the target file, not the source file, and as long as we need to copy in the same folder, we still have this problem?

@Berdir I think what #97 is saying is that sites/default/default.settings.php is used as a reference and we would instead refer people to core/assets/scaffold/files/default.settings.php. You would still have sites/default/settings.php which you could customise, and we would never need to copy default.settings.php in to sites/default.

re #99

Doesn't using scaffold files directly pretty much defeat the point of having scaffold files in the first place, aka the ability to customize them.

I don't think that is only purpose the scaffolding.

From https://www.drupal.org/docs/develop/using-composer/using-drupals-compose... →

The purpose of scaffolding files is to allow Drupal sites to be fully managed by Composer, and still allow individual asset files to be placed in arbitrary locations. The goal of doing this is to enable a properly configured composer template to produce a file layout that exactly matches the file layout of a Drupal 8.7.x and earlier tarball distribution.

But I agree my idea in #97 defeats the purpose of site/default/default.* files being in core/assets/scaffold/files because I was proposing never copying these files into another location at all.

So I guess the question is do people really need files like default.setitngs.php to be customized? I am not sure sites want customize the file used as a starting point that settings.php is made from by the installer.

I think that default.settngs.php is scaffold file at all for legacy reasons and the other purpose in the doc for the scaffold is

Other file layouts are also possible; for example, a project layout very similar to the now deprecated drupal-composer/drupal-project template is provided as part of the Drupal Recommended Project composer template.

To customize the contents of default.setitngs.php so the installer would use the customize version of default.settngs.php to make settings.php the project would have to add "[web-root]/sites/default/default.settings.php": "custom_scaffold_files/default.settings.php",. Is that really a use case that is important to support?

The scaffold does allow moving files into different locations but if anyone customized the location of default.setitngs.php to anywhere else other than sites/default I think the installer would break because install_check_requirements has this relative location hardcoded.

So it is possible that people to customize the contents of default.setitngs.php but seems unlikely. It doesn't seem possible that you can change the location of default.setitngs.php because of install_check_requirements.

I think default.setitngs.php right now is a scaffold file at all because we want it to be in same directory as settings.php should be in and the scaffold is how we move files that are in different locations with different composer project layouts. But is that really a necessity default.settings.php live at sites/default? It does certainly make things more difficult that the scaffold has to write to sites/default and we also automatically harden that location. Isn't the more common case the installer itself uses default.setitngs.php to make settings.php and in that case we tell the installer it is anywhere.

So why not not just make new location like core/assets/defaults that would contain default.settings.php and default.services.yml? Then just update the installer to use the new location and update core/install.txt

If there are BC concerns for Drupal 10 we could just use core/assets/scaffold/files/default.settings.php for now but remove from being moved and in Drupal 11 use core/assets/defaults/default.settings.php

A related idea: if sites/default/default.*:

1. are never used (as in executed)
2. must always be up-to-date
3. are present in that specific location only for ease-of-use

… then why don't we symlink them from core/assets/scaffold/files/default.*? That way, they're always up-to-date, and we never have to deal with permissions ever again.

🤔

Although it appears that symlinks do not have their own permissions (they automatically inherit the permissions of the target), there is one exception apparently: macOS. Investigation needed.

So why not not just make new location like core/assets/defaults that would contain default.settings.php and default.services.yml? Then just update the installer to use the new location and update core/install.txt

This sure sounds like a simple and logical solution to me 😄 Probably the simplest solution possible.

Curious to hear what the concerns would be for that proposal!

> So I guess the question is do people really need files like default.setitngs.php to be customized?

I doubt it's done often, but I can think of use cases, some kind of distribution/template could allow to customize default.settings.php, but I guess then you could also just provide a settings.php in the first place then.

default.services.yml is not copied by default though, it's really just an example, so not sure if that's easy to find then.

Could indeed make sense to do that, but no idea in what form that's a BC break and when we are allowed to make that change, could also have quite a bit impact on documentation that needs to updated.

It would IMHO be easier to just rip out the logic around the permission check entirely, that was also proposed in the D7 backport issue of the skip_permissions_hardening setting IIRC.

Another solution:

Have Composer scaffold core/assets/defaults/default.settings.php to sites/templates/default.settings.php.

This new sites/templates folder would not be executable and contain only templates or examples files.

We could use examples instead, too. So either:

sites/templates/default.settings.php
sites/examples/default.settings.php

Honestly, it'd be great if we didn't have to have sites/default/default.settings.php as a requirement. No symlink, so copy. It makes running Drupal out of the vendor directory that much easier.

I still think it makes sense to have Composer chmod 775 sites/default.

templates approach will lead to outdated files and more errors on updates because user's will forget to update files

templates approach will lead to outdated files and more errors on updates because user's will forget to update files

I would expect Composer to scaffold the files inside templates. Thus all those files, managed by Composer scaffold, inside the template folder, will always be up-to-date.

I suggest to scaffold the default.settings.php file in sites/templates instead of sites/default. Same mechanism. So no outdated files.

Just a clarification:

do we actually use the exact page docroot/ in:

"drupal-scaffold": {
"locations": {
"web-root": "docroot/"
},
"file-mapping": {
"[web-root]/sites/default.services.yml": false,
"[web-root]/sites/default.settings.php": false
}
},

Or is that meant to be a placeholder for the current document root of your installation, which for me is web/. I've tried both and using docroot/ creates a whole new drupal installation in a new folder called docroot, which didn't seem right but the error stopped. I also tried I using web/ which seems right but I still get the "could not delete" error messages .

+1 on #103. Not sure if the templates or the defaults terminology fits better.
I think a default is something, that's used "by default" and is kind of required and "active"
While a template is best-practice that can be used but is not actively used

The risk of #107 is true, but the opposite is overwriting modifications?
Best might be to have an inheritance logic, but that may also lead to collisions with overwrites / customizations.
No Silver Bullet. What's best and how can we figure it out?

This is critical, and this issue is getting close to its four year birthday.

Following the officially recommended method of Updating Drupal core via Composer → to deploy a new composer.lock file on the production server with composer install can get you a critical error, and a broken web site, until you understand what the problem is (finding this page), implementing the workaround here, finding the commands to relax permissions, complete the update, and then resetting file permissions and ownership → again, hoping it doesn't happen next time.

What should be a boring and uneventful task becomes a painful experience. The next update will be dreaded.

+1 for moving sites/default/default.settings.php elsewhere as @tedbowman and @mglaman suggest, if that solves the problem. If it also makes work on Automatic Updates easier, that's another strong argument for this proposal.

Another solution is to remove the whole default.settings.php file altogether and document the default values in the README.md file.

But if we want to keep that file, we can rename it to default.settings.txt so that PHP will not try to execute it.

Hi xmacinfo, note that renaming a php file to txt, makes it readable which is imho a lot worse.

some solutions could be :
- scaffolding in a dedicated directory that would remain writable eg. sites/examples/ as suggested in #111 (proper name might be discussed)
- not scaffolding it (eventually add a composer command to copy it when needed, or displaying a note telling where core example assets can be copied from)

I think these 2 files are usually should be copied and then updated before any integration and any modification to these files can cause permission hardening.

sites/default/default.services.yml
sites/default/default.settings.php

I came across this problem while moving from 9.5.x to 10.x and i feel + 1 to modify the scaffolding process in a manner that if the permission of incoming version differs from what is locally present try to do the below:

• Store the existing permission and do a chmod 775 on sites/default
• replace or overwrite based on the scaffolding process set in composer.json
• Then restore the existing permission from bullet 1

Else fallback to the default behaviour.

I'm coming from https://www.drupal.org/drupalorg/blog/introducing-the-bounty-program → and just read everything so I'd like to make a summary of the current state of this issue. First of all, I'm somewhat "ignoring" the comments and patches before #67 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work as they happened before 3103090 → was implemented.

First of all, these are the scenarios that could cause this issue to happen:

1. The site builder modifies the files ( #67 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work )
2. The site builder removes the files ( #67 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work )
3. Core updates the files ( #92 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work )

These are the workarounds:

1. composer.json changes

Documentation: https://www.drupal.org/docs/develop/using-composer/using-drupals-compose... →
Code:

"extra": {
  drupal-scaffold": {
    "file-mapping": {
      "[web-root]/sites/default/default.services.yml": false,
      "[web-root]/sites/default/default.settings.php": false
    }
  }
}

See #73 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work .

The following are the suggestions to fix the issue:

1. Trying harder

Suggested by greg.1.anderson → on #67 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work :

a simpler strategy of simply trying harder would be appropriate. i.e. the scaffold plugin could always try to make a write-protected file writable first, and only fail if it does not have permission to do that.

My thoughts

It's a good idea, but we risk leaving the folder/file writable if the composer command fails in the middle of the process, I know it's unlikely, but imagine a scenario where we change the folder/file permission to writable and composer fails because of OOM before reverting that change, if you try to run the command again it will just succeed but it won't "unharden" (change the permissions back to what they were before) as it won't know what permissions they were before.

2. Optional scaffold files

Suggested by greg.1.anderson → on #67 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work :

If a scaffold file is optional, then any attempt to modify the file that fails becomes a warning instead of an error. This idea could also be done in a follow-on issue.

#95 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work also metions the same solution.

My thoughts

I like the idea and I think it also links to the next suggestion.

3. Don't even try

Suggested by tedbow → on #97 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work :
I can't find a quote so here's a summary: He suggests rethinking the need for scaffolding sites/default/default.* files to avoid folder permission issues and facilitate the Automatic Updates initiative.

TLDR: Don't scaffold these files at all.

My thoughts

I like this idea. There is a concern raised about this though which is cases where the Drupal installation is actually using that file.

4. Symlink

Suggested by Wim Leers → on #102 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work :

why don't we symlink them from core/assets/scaffold/files/default.*? That way, they're always up-to-date, and we never have to deal with permissions ever again.

My thoughts

I like this idea but the same concern from the previous suggestion applies here I believe, if an installation is using the file how is it gonna update the contents of it if it's not actually a file, only a link? Developers have the option to append files → as part of the scaffolding process as well.

5. Templates folder

Suggested by xmacinfo → on #105 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work :

Have Composer scaffold core/assets/defaults/default.settings.php to sites/templates/default.settings.php.

My thoughts

We need to remember that the sites directory is used by multisites Drupal installations and that would mean if there is an installation called "examples" or "templates" they would now have new files into their folder "out of nowhere".

6. md or txt

Suggested by xmacinfo → on #112 🐛 Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable. Needs work :

remove the whole default.settings.php file altogether and document the default values in the README.md file

if we want to keep that file, we can rename it to default.settings.txt so that PHP will not try to execute it.

we need Composer Scaffold to store that file outside of the default protected folder.

My thoughts

Someone else raised the concern about using those two extensions that they might become available to end-users, as webservers are usually configured to block .php files but to allow .md and .txt files, so this comes with security risks attached.

My Suggestion

Do what suggestion number 3 is saying, but because it's possibly not BC for some use case scenarios, do what suggestion number 2 is saying for now until we release a new major version (Drupal 11?), have a list of files that are "optional" (naming to be discussed), try to modify them, but don't fail the whole process, only display a deprecation warning linking to a piece of documentation of why this warning is being displayed and what can be done to get rid of it.
This means we would have to update all the references to these files, such as the core/Install.txt file and others.

Another potential solution would be to simply do suggestion 2 as part of this issue, and the rest as part of a follow-up. Just to make sure we resolve the actual problem of devs not being able to run composer commands as soon as possible.

I tried to reproduce it on 10.x branch by doing the below changes but the issue is not happening anymore:

• I intentionally added chmod 444 to sites/default/default.settings.php file while on a 9.5.x branch
• Moved to Drupal 10.x branch and run composer update but what i can see is that the 444 got converted to 644

I think this ReplaceOp::process() method is managing the files in a correct manner :

/**
   * {@inheritdoc}
   */
  public function process(ScaffoldFilePath $destination, IOInterface $io, ScaffoldOptions $options) {
    $fs = new Filesystem();
    $destination_path = $destination->fullPath();
    // Do nothing if overwrite is 'false' and a file already exists at the
    // destination.
    if ($this->overwrite === FALSE && file_exists($destination_path)) {
      $interpolator = $destination->getInterpolator();
      $io->write($interpolator->interpolate("  - Skip <info>[dest-rel-path]</info> because it already exists and overwrite is <comment>false</comment>."));
      return new ScaffoldResult($destination, FALSE);
    }

    // Get rid of the destination if it exists, and make sure that
    // the directory where it's going to be placed exists.
    $fs->remove($destination_path);
    $fs->ensureDirectoryExists(dirname($destination_path));
    if ($options->symlink()) {
      return $this->symlinkScaffold($destination, $io);
    }
    return $this->copyScaffold($destination, $io);
  }

The Needs Review Queue Bot → tested this issue.

While you are making the above changes, we recommend that you convert this patch to a merge request → . Merge requests are preferred over patches. Be sure to hide the old patch files as well. (Converting an issue to a merge request without other contributions to the issue will not receive credit.)

vakulrai → closed merge request !6312

vakulrai → reopened merge request !6312

vakulrai → changed the visibility of the branch 3091285-composer-scaffolding-fails-preprocess to hidden.

11.x branch has merge conflicts can someone rebase it.

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Fixed failing test cases.

The Needs Review Queue Bot → tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Updating Current workaround, to match the default paths:

"drupal-scaffold": {
    "locations": {
        "web-root": "web/"
    },
    "file-mapping": {
        "[web-root]/sites/default/default.services.yml": false,
        "[web-root]/sites/default/default.settings.php": false
    }
},