Hosting environment (e.g. cPanel) may add additional files (including symlinks) to the project, which breaks AU

This is at a minimum postponed on 📌 Limit trusted Composer plugins to a known list, allow user to add more Fixed and 📌 Exclude unknown paths in project base: only allow vendor + web root + whatever drupal/core-composer-scaffold allows Fixed . Clarifying.

📌 Limit trusted Composer plugins to a known list, allow user to add more Fixed is fixed.

📌 Exclude unknown paths in project base: only allow vendor + web root + whatever drupal/core-composer-scaffold allows Fixed is the only thing that remains.

📌 Exclude unknown paths in project base: only allow vendor + web root + whatever drupal/core-composer-scaffold allows Fixed landed, which means this should be fixed now!

As soon as @tedbow has tagged a new 2.7 release, we should ask @chike to update and report back 😊

I am waiting to do that.

I have been following the work. It's amazing.

Cool to know you've been following along! 😊

Yes I have been following along and admiring how the work was going. I wish I knew enough programming to contribute, but maybe someday....maybe.

@chike the 8.x-2.7 release is out if you want to try it

Per #24 — over to you, @chike! 😊

Updated to 2.7 and site throws WSOD with message,

TypeError: array_map(): Argument #2 ($array) must be of type array, null given in array_map() (line 130 of /home/project-root/public_html/modules/contrib/automatic_updates/package_manager/src/Validator/ComposerPluginsValidator.php)

@chike thanks for testing

Sorry about that I think that is our fault. I created 🐛 No update hook for additional_trusted_composer_plugins setting Closed: works as designed which I think is the problem

I credited you on that issue

#26: you did not run yoursite.com/update.php — i.e. you forgot to apply database updates 😅

I forgot to update the database, that solved the issue. I am sorry that caused you unnecessary trouble.

But the symlinks are still there.

It's a long list of them,

user@xxxxxx [~]# find . -type l
./.cpanel/ea-php-cli/.ea-php-cli.cache
./.cpanel/ea-php-cli/composer-sal/.ea-php-cli.cache
./.cpanel/ea-php-cli/drush/.ea-php-cli.cache
./.cpanel/ea-php-cli/drush/vendor/drush/drush/.ea-php-cli.cache
./.cpanel/ea-php-cli/public_html/.ea-php-cli.cache
./.cpanel/ea-php-cli/vendor/bin/.ea-php-cli.cache
./.cpanel/ea-php-cli/vendor/drush/drush/.ea-php-cli.cache
./.cpanel/ea-php-cli/composer/.ea-php-cli.cache
./www
./access-logs
./.cphorde/meta/latest
./etc/mywebsite.com/info.rcube.db.latest
./mail/.info@mywebsite_com
./mail/.marketing@mywebsite_com
./vendor/grasmash/yaml-expander/scenarios/symfony2/src
./vendor/grasmash/yaml-expander/scenarios/symfony2/tests
./vendor/grasmash/yaml-expander/scenarios/symfony4/src
./vendor/grasmash/yaml-expander/scenarios/symfony4/tests
./vendor/drush/drush/docs/drush_logo-black.png
./vendor/drush/drush/docs/misc/icon_PhpStorm.png
./vendor/drush/drush/docs/img/favicon.ico
./vendor/drush/drush/docs/contribute/CONTRIBUTING.md
./.cagefs/opt/alt/php81/link/conf
./.cagefs/opt/alt/php74/link/conf
./.cagefs/opt/alt/php52/link/conf
./.cagefs/opt/alt/php80/link/conf
./.cagefs/opt/alt/php55/link/conf
./.cagefs/opt/alt/php56/link/conf
./.cagefs/opt/alt/php82/link/conf
./.cagefs/opt/alt/php54/link/conf
./.cagefs/opt/alt/php71/link/conf
./.cagefs/opt/alt/php70/link/conf
./.cagefs/opt/alt/php73/link/conf
./.cagefs/opt/alt/php44/link/conf
./.cagefs/opt/alt/php51/link/conf
./.cagefs/opt/alt/php72/link/conf
./.cagefs/opt/alt/php53/link/conf
./.cagefs/tmp/mysql.sock
./.cagefs/tmp/.s.PGSQL.5432

The symlinks won't disappear. Automatic Updates/Package Manager didn't put them there!

What matters, is that these no longer prevent you from using Automatic Updates. If you visit the status report, do you still get error messages for Automatic Updates/Package Manager?

Besides #30 if you goto the update form at `/admin/reports/updates/update` is the update button available? Or course if you are the on the latest patch version of you drupal core minor then there would be no version to update to.

If the Update button is not there does it show you any errors?

For #30 update readiness checks is still failing,

Update readiness checks
Your site does not pass some readiness checks for automatic updates. It cannot be automatically updated until further action is performed.
The active directory at "/home/project-root" contains symlinks, which is not supported. The first one is "/home/project-root/.cagefs/opt/alt/php44/link/conf". See the help page for information on how to resolve the problem.

For #31 the update button is not visible and I have pending core updates. I get the message,

Your site does not pass some readiness checks for automatic updates. It cannot be automatically updated until further action is performed.
The active directory at "/home/project-root" contains symlinks, which is not supported. The first one is "/home/project-root/.cagefs/opt/alt/php44/link/conf". See the help page for information on how to resolve the problem.

Hm … 🤔

This suggests that 📌 Exclude unknown paths in project base: only allow vendor + web root + whatever drupal/core-composer-scaffold allows Fixed is not actually finding all the files in the project root … I think we should create a test that simulates @chike's environment, which roughly looks like this:

home/
  project-root/
    .cagefs/
      symlink-here
    vendor/
      …
    www/
      …

→ UnknownPathExcluder should add /home/project-root/.cagefs/symlink-here to the collection of ignored paths!

Assigning to @kunal.sachdev since he did 99% of the work on 📌 Exclude unknown paths in project base: only allow vendor + web root + whatever drupal/core-composer-scaffold allows Fixed , so he'll be the most proficient at writing this test 😊

If you want to test on a shared hosting server you can purchase a shared hosting plan from Greengeeks (which I am using) Once activated you will find that your web-root is a folder called public_html (this is what servers managed with cPanel software always have as web-root by default) so you should make a modified install to place Drupal inside public_html as against the default web folder. The folder containing public_html will then be your project-root. The /home is part of the absolute path for the host account but never accessible.

kunal.sachdev → made their first commit to this issue’s fork.

@kunal.sachdev Next time, please push up a test-only commit first, so we can see it fail. Then we can see it pass after you push the fix. That makes it crystal clear that the test catches the problem and the fix fixes it 🤓

I did that locally: I reverted the fix, kept only the test additions, and it indeed fails:

PHPUnit 9.5.26 by Sebastian Bergmann and contributors.

Testing /Users/wim.leers/core/modules/contrib/automatic_updates/package_manager/tests/src/Kernel/PathExcluder
...F...F                                                            8 / 8 (100%)

Time: 00:11.159, Memory: 10.00 MB

There were 2 failures:

1) Drupal\Tests\package_manager\Kernel\PathExcluder\UnknownPathExcluderTest::testUnknownPath with data set "unknown hidden file where web and project root different" (true, null, array('.unknown_file'))
Failed asserting that file "/private/tmp/package_manager_testing_roottest10647682/stage/3ou-WDmPdCroz5R55CbNm_IAtJGxK_GVv-8bR87nElU/.unknown_file" does not exist.

/Users/wim.leers/core/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:122
/Users/wim.leers/core/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/Users/wim.leers/core/modules/contrib/automatic_updates/package_manager/tests/src/Kernel/PathExcluder/UnknownPathExcluderTest.php:164
/Users/wim.leers/core/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

2) Drupal\Tests\package_manager\Kernel\PathExcluder\UnknownPathExcluderTest::testUnknownPath with data set "unknown hidden directory where web and project root different" (true, '.unknown_dir', array('.unknown_dir/unknown_dir.README.md', '.unknown_dir/unknown_file.txt'))
Failed asserting that file "/private/tmp/package_manager_testing_roottest57665865/stage/0ODADlWYhJAdEqp1g24Z82pPyhUfvCaKcDLkQiRMXso/.unknown_dir/unknown_dir.README.md" does not exist.

/Users/wim.leers/core/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:122
/Users/wim.leers/core/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/Users/wim.leers/core/modules/contrib/automatic_updates/package_manager/tests/src/Kernel/PathExcluder/UnknownPathExcluderTest.php:164
/Users/wim.leers/core/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

FAILURES!
Tests: 8, Assertions: 47, Failures: 2.
Process finished with exit code 1

I applied MR 709 patch, cleared caches, reran readiness checks and yet same thing, site does not pass.

Back to the ol' drawing board, based on #39. 😭 Keep fighting the good fight! 💪

@kunal.sachdev I would suggest making local composer project and adding some symlinks like is describing in this issue. It would be good to see if you saw the same thing outside of tests.

So I made a local composer project with Drupal core 10.0.0 and created a symlink. I updated my site to 10.0.3 successfully using automatic updates.

Could you try with web-root named differently from web.

#42: Can you show exactly what the directory structure was that you tested with? That's easy to achieve if you use tree (brew install tree):

tree -a -L 2 -F -n --filesfirst

should give you something like:

$ tree -a -L 2 -F -n --filesfirst
./
├── .DS_Store
├── .csslintrc
├── .editorconfig
├── .eslintignore
├── .eslintrc.json
├── .gitattributes
<snip>
├── .git/
<snip>
├── .idea/
<snip>
├── composer/
<snip>
├── core/
│   ├── .cspell.json
<snip>

Use that to make it easy for us to understand what exactly you're testing :)

That, plus find . -type l of course.

 $ tree -a -L 2 -F -n --filesfirst
./
├── .editorconfig
├── .gitattributes
├── composer.json
├── composer.lock
├── symlink -> composer.json
├── vendor/
│   ├── autoload.php
│   ├── asm89/
│   ├── bin/
│   ├── composer/
│   ├── doctrine/
│   ├── drupal/
│   ├── egulias/
│   ├── guzzlehttp/
│   ├── justinrainbow/
│   ├── masterminds/
│   ├── pear/
│   ├── php-tuf/
│   ├── psr/
│   ├── ralouphie/
│   ├── react/
│   ├── seld/
│   ├── symfony/
│   └── twig/
└── web/
    ├── .csslintrc
    ├── .eslintignore
    ├── .eslintrc.json
    ├── .ht.router.php
    ├── .htaccess
    ├── INSTALL.txt
    ├── README.md
    ├── autoload.php
    ├── example.gitignore
    ├── index.php
    ├── robots.txt
    ├── update.php
    ├── web.config
    ├── core/
    ├── modules/
    ├── profiles/
    ├── sites/
    └── themes/

25 directories, 19 files

and

$ find . -type l                 
./symlink

@chike can you upload the composer.json file at the root of your project? Thanks

Here.

So I tried it again with the following directory structure :

./
├── .editorconfig
├── .gitattributes
├── composer.json
├── composer.lock
├── .cagefs/
│   └── opt/
├── .idea/
│   ├── .gitignore
│   ├── modules.xml
│   ├── my-test-project.iml
│   └── workspace.xml
├── public_html/
│   ├── .csslintrc
│   ├── .eslintignore
│   ├── .eslintrc.json
│   ├── .ht.router.php
│   ├── .htaccess
│   ├── INSTALL.txt
│   ├── README.md
│   ├── autoload.php
│   ├── example.gitignore
│   ├── index.php
│   ├── robots.txt
│   ├── update.php
│   ├── web.config
│   ├── core/
│   ├── modules/
│   ├── profiles/
│   ├── sites/
│   └── themes/
└── vendor/
    ├── autoload.php
    ├── asm89/
    ├── bin/
    ├── composer/
    ├── doctrine/
    ├── drupal/
    ├── egulias/
    ├── guzzlehttp/
    ├── justinrainbow/
    ├── masterminds/
    ├── pear/
    ├── php-tuf/
    ├── psr/
    ├── ralouphie/
    ├── react/
    ├── seld/
    ├── symfony/
    └── twig/

28 directories, 22 files

and

$ find . -type l                                  
./.cagefs/opt/alt/php44/link/conf

and
I updated my site to 10.0.3 successfully using automatic updates.

@kunal.sachdev's manual testing in #48 suggests that it should work fine for @chike too … 🤔

@chike In #39 you said you had applied the patch for this MR. Are you sure it was truly applied? A small mistake is easy to make :) How did you apply it exactly? Using cweagans/composer-patches or some other mechanism?

In #47 🐛 Hosting environment (e.g. cPanel) may add additional files (including symlinks) to the project, which breaks AU Postponed: needs info I posted my composer.json so you would see how I added the MR patch.

I tried AU in another D9 site and got this message:

Your site does not pass some readiness checks for automatic updates. It cannot be automatically updated until further action is performed.
The active directory at "/home/project-root" contains symlinks, which is not supported. The first one is "/home/project-root/vendor/drush/drush/docs/contribute/CONTRIBUTING.md". See the help page for information on how to resolve the problem.
An unsupported Composer plugin was detected.
oomphinc/composer-installers-extender

On this second site I am using Asset Packagist to install libraries and wanted the libraries installed in libraries folder and not vendor so I added https://github.com/oomphinc/composer-installers-extender to be able to do that. This plugin should be popular so good for AU to exclude it.

Then I tried on a new D10 site I installed on a different shared hosting company and on this site I got no mention of symlinks, only one blocker as in the message I got;

Your site does not pass some readiness checks for automatic updates. It cannot be automatically updated until further action is performed.
Unable to collect ignored paths, therefore can't perform status checks.
RecursiveDirectoryIterator::__construct(/home/project-root/.nc_plugin): Failed to open directory: Permission denied

I listed the symlinks on the site and got:

[superuser@server180 ~]$ find . -type l
./mail/.info@website1_com
./mail/.chike@website2_com
./mail/.admin@website1_com
./demo1/vendor/grasmash/yaml-expander/scenarios/symfony2/tests
./demo1/vendor/grasmash/yaml-expander/scenarios/symfony2/src
./demo1/vendor/grasmash/yaml-expander/scenarios/symfony4/tests
./demo1/vendor/grasmash/yaml-expander/scenarios/symfony4/src
./demo1/vendor/bin/dcg
./demo1/vendor/bin/psysh
./demo1/vendor/bin/var-dump-server
./demo1/vendor/bin/release
./demo1/vendor/bin/robo
./demo1/vendor/bin/php-parse
./demo1/vendor/bin/drush
./demo1/vendor/consolidation/log/.scenarios.lock/symfony2/tests
./demo1/vendor/consolidation/log/.scenarios.lock/symfony2/src
./demo1/vendor/consolidation/log/.scenarios.lock/symfony4/tests
./demo1/vendor/consolidation/log/.scenarios.lock/symfony4/src
./demo1/vendor/consolidation/site-alias/.scenarios.lock/phpunit5/tests
./demo1/vendor/consolidation/site-alias/.scenarios.lock/phpunit5/src
./demo1/vendor/drush/drush/docs/misc/icon_PhpStorm.png
./demo1/vendor/drush/drush/docs/img/favicon.ico
./demo1/vendor/drush/drush/docs/index.md
./demo1/vendor/drush/drush/docs/contribute/CONTRIBUTING.md
./demo1/vendor/drush/drush/docs/drush_logo-black.png
./.cagefs/opt/alt/php53/link/conf
./.cagefs/opt/alt/php70/link/conf
./.cagefs/opt/alt/php52/link/conf
./.cagefs/opt/alt/php72/link/conf
./.cagefs/opt/alt/php71/link/conf
./.cagefs/opt/alt/php74/link/conf
./.cagefs/opt/alt/php54/link/conf
./.cagefs/opt/alt/php56/link/conf
./.cagefs/opt/alt/php81/link/conf
./.cagefs/opt/alt/php55/link/conf
./.cagefs/opt/alt/php73/link/conf
./.cagefs/opt/alt/php80/link/conf
./.cagefs/tmp/mysql.sock
./.cagefs/tmp/.s.PGSQL.5432
./access-logs
./vendor/drush/drush/docs/misc/icon_PhpStorm.png
./vendor/drush/drush/docs/img/favicon.ico
./vendor/drush/drush/docs/contribute/CONTRIBUTING.md
./vendor/drush/drush/docs/drush_logo-black.png
find: `./.nc_plugin': Permission denied
./www

Note this line: find: `./.nc_plugin': Permission denied. From little Googling it's like .nc_plugin is troublesome. Here is info on it as on the website of the host company, Namecheap where I uploaded the D10 site, https://www.namecheap.com/support/knowledgebase/article.aspx/10294/29/ho...

So on two D9 sites I got symlinks blocking AU and on one D10 site it didn't mention symlinks

@kunal.sachdev could you make a test using D9?

With 📌 Add symlink support to Composer Stager 2.0, require that version, and simplify UX & tests accordingly Fixed committed, the 3.0.x branch should simply not complain about this at all!