- Issue created by @ergonlogic
- 🇨🇦Canada ergonlogic Montréal, Québec 🇨🇦
Tested on a small personal project. Results:
- Packages: 15
- TUF metadata: 2.9M
- 🇨🇦Canada ergonlogic Montréal, Québec 🇨🇦
Tested on a second small side-project:
Packages: 19
TUF metadata: 2.8MI'm planning to test on a bunch of different projects. To make this more efficient, I put together a couple simple scripts that just run the steps outlined in the issue summary.
- 🇨🇦Canada ergonlogic Montréal, Québec 🇨🇦
Report from a medium-sized application:
Packages: 53
TUF metadata: 3.6M - 🇪🇸Spain fjgarlin
When running this
composer tuf:protect https://packagist-signed.drupalcode.org
I got the following:Authenticity of packages from https://packagist-signed.drupalcode.org are not verified by TUF. [red] There are no commands defined in the "tuf" namespace. [/red] Composer [tuf:protect https://packagist-signed.drupalcode.org] failed, composer command failed: exit status 1. stderr=
I edited the
composer.json
file manually to have:"drupal-core": { "type": "composer", "url": "https://packagist-signed.drupalcode.org", "tuf": true },
And then I could continue.
Report:
- Packages: 32
- TUF metadata: 3.0M - 🇪🇸Spain fjgarlin
I re-did the whole thing from scratch on the above project and I did not get that small error, so all the steps worked directly.
- 🇨🇦Canada ergonlogic Montréal, Québec 🇨🇦
FYI, you may occasionally see an error like:
The 'bin_c50-c53' contents does not match hash 'sha256' specified in the 'snapshot' metadata.
Sometimes this is transitory, and will fix itself in a matter of seconds. Re-running `composer update` will likely succeed. If not, but the 'bin_c50-c53' part changes, wait a minute or two and try again. This is a symptom of temporary inconsistency during metadata updates on the server-side. This is being addressed upstream in rugged/rugged#208: Ensure metadata consistency.
On the other hand, this kind of error can also persist longer. If re-running `composer update` show no change in the 'bin_c50-c53' part of the error message, this likely indicates that the TUF server has "stalled" during a metadata update. This will generally require intervention by a TUF server administrator. This is being addressed upstream in rugged/rugged#204: Add a monitor task to clean up stuck targets.
We expect this to stabilize as we address the upstream issues. However, if you do encounter such an error, it is worthwhile to report it here, in this ticket.
- 🇬🇷Greece vensires
I did the process and it didn't work initially. As @fjgarlin said in #8, only after I did the process from start a second time it did work. Maybe we should have first executed the code to download the initial root metadata for the repos and only then try to add the signed repository for Drupal core and enable TUF protection for it? I mean, it doesn't seem like an issue of the Packaging component but more like an incorrect order of the commands as described in the issue summary here.
Packages: 77 TUF metadata: 4.5M
- 🇺🇸United States papagrande US West Coast
I also had problems with installation and now get out of memory errors when running
composer up
in both ddev and on my MacOS host.
PHP Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/php-tuf/php-tuf/src/CanonicalJsonTrait.php on line 46
FWIW, here is my data on one client's installation after several attempts:
Packages: 118
TUF metadata: 5.3M - 🇨🇦Canada nickdickinsonwilde Victoria, BC (T'So-uke lands)
Hit an immediate hard failure - opened PR https://github.com/php-tuf/composer-integration/pull/126
With applying that, get:https://packagist-signed.drupalcode.org could not be fully loaded (Maximum allowed download size reached. Downloaded 1165080 of allowed 1165080 bytes), package information was loaded from the local cache and may be out of date
(this continues after many runs)
Packages: 81
TUF metadata: 5.4mbFailing with "The 'bin_d9c-d9f' contents does not match hash 'sha256' specified in the 'snapshot' metadata." - for a while.
- 🇬🇧United Kingdom catch
I haven't done the steps yet but #10 suggests this needs work one way or the other.
- 🇺🇸United States drumm NY, US
On the other hand, this kind of error can also persist longer. If re-running `composer update` show no change in the 'bin_c50-c53' part of the error message, this likely indicates that the TUF server has "stalled" during a metadata update. This will generally require intervention by a TUF server administrator. This is being addressed upstream in rugged/rugged#204: Add a monitor task to clean up stuck targets.
https://gitlab.com/drupal-infrastructure/package-signing/rugged-metrics now alerts us when targets are stuck, but it does require manual intervention to resolve. Once https://gitlab.com/rugged/rugged/-/issues/204 is deployed, this will be more automatic.
Most of the issues we have seen are actually directories stuck in the building phase, before Rugged is involved. There is also some chance of disruption on Tuesdays during the regular AWS maintenance window for the managed queue.
PHP Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/php-tuf/php-tuf/src/CanonicalJsonTrait.php on line 46
1.6MB is not a lot, especially when Drupal will require 64MB. Regardless, the php-tuf client will need to be better at detecting constrained memory situations, and keeping memory usage low. We do want to be judicious with memory use, since the hope is to expand TUF signing to the wider PHP ecosystem.
https://packagist-signed.drupalcode.org could not be fully loaded (Maximum allowed download size reached. Downloaded 1165080 of allowed 1165080 bytes), package information was loaded from the local cache and may be out of date
I think the Composer plugin client will need some error handling improved. This suggests something is amiss server-side, good chance cached at the CDN, but I don’t know what it might be. Running with
-vvv
might help point to which URL might be the issue.Failing with "The 'bin_d9c-d9f' contents does not match hash 'sha256' specified in the 'snapshot' metadata." - for a while.
This looks like it is resolved now:
$ curl --silent https://packagist-signed.drupalcode.org/metadata/snapshot.json | grep -A2 bin_d9c-d9f "bin_d9c-d9f.json": { "hashes": { "sha256": "bf9500067989a47152416e9dde6b7c059286657d080f2fd526949e462d5c0172", $ curl --silent https://packagist-signed.drupalcode.org/metadata/bin_d9c-d9f.json | sha256sum bf9500067989a47152416e9dde6b7c059286657d080f2fd526949e462d5c0172 -
Right now we are permanently caching at the CDN and purging when changes happen. I’ve seen caching be enough of an issue that I think I will put a 5 or 10 minute lifetime on the cache at the CDN. So issues like this should self-heal after a few minutes.
- 🇺🇸United States drumm NY, US
Opened a few issues to track what I mentioned in my last comment:
- Improve resilience of uploading building directories https://gitlab.com/drupal-infrastructure/package-signing/packagist-signe... this is an operational improvement. When a building directory is stuck, rugged continues as usual
- Improve handling of low-memory situations https://github.com/php-tuf/composer-integration/issues/127
- Improve error messages to show which HTTP request is the source of errors https://github.com/php-tuf/composer-integration/issues/128
- The fallback TTL of 1 hour was being used for TUF metadata at the CDN. It is now reduced to 10 minutes
I moved “To provide the root of trust for TUF verifications, download the initial root metadata for these two repos:” up in the issue summary, which should make the setup steps run more smoothly. While there are things to improve, we do need more real-world testing to see if the improvements have been effective, so setting back to needs review.
- 🇦🇺Australia skyejohnson Sunshine Coast, Australia
I also got the same result as #7 and @fjgarlin's notes fixed it.
I had to increase my php memory limit to runcomposer -vv update
Report from my medium D11 site:
- Packages: 31
- TUF metadata: 3.9M Same here as the previous tests, some memory issues, plus the order of commands.
My results:- Packages: 89
- TUF metadata: 6.6M- 🇺🇸United States drumm NY, US
Putting my edits to address #7 back in the issue summary.
- 🇺🇸United States drumm NY, US
skyejohnson & kyriazo - what was your PHP memory limit that you hit when you started, and how much did you raise it to succeed?
- 🇦🇺Australia skyejohnson Sunshine Coast, Australia
@drumm I just set it to -1 tbh
- 🇺🇸United States cmlara
Using the time utility (make sure its the binary one not the bash built-in) would also probably be useful to log memory usage.
I put this into a contrib module lab (in no way representative of an actual deployment, just 10 packages of drupal/core using 3.0 mb in the tuf folder to develop a single contrib module)
With the tuf plugin:
$ /usr/bin//time composer -vv update 22.15user 3.12system 0:50.02elapsed 50%CPU (0avgtext+0avgdata 975440maxresident)k 88inputs+297600outputs (0major+270631minor)pagefaults 0swaps
Without the tuff plugin:
$ /usr/bin//time composer -vv update 10.19user 3.09system 0:20.90elapsed 63%CPU (0avgtext+0avgdata 280376maxresident)k 0inputs+285856outputs (0major+90243minor)pagefaults 0swaps
Max resident memory consumption increase, 695mb, a 3.4x increase..
(Jut to be clear #14 mentions 1.6Mb, the OOM report is actually 1.6Gb)COMPOSER_MEMORY_LIMIT=960m - succeed with tuff (succeeds by less than 64mb)
COMPOSER_MEMORY_LIMIT=256m - succeed without tuf (by less than 10mb) - 🇺🇸United States drumm NY, US
I misread the conversion here:
PHP Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/php-tuf/php-tuf/src/CanonicalJsonTrait.php on line 46
1.6MB is not a lot, especially when Drupal will require 64MB. Regardless, the php-tuf client will need to be better at detecting constrained memory situations, and keeping memory usage low. We do want to be judicious with memory use, since the hope is to expand TUF signing to the wider PHP ecosystem.
This is actually 1.6GB, which is already quite a high memory limit. Along with #21, it does look like the client seems to have a memory usage problem. So https://github.com/php-tuf/composer-integration/issues/127 is looking like the biggest blocker to getting this closer to core.
- 🇦🇺Australia fubarhouse
Neil brought this to my attention, so I've been taking a quick look at this as part of DrupalCon contribution day in Singapore.
I've used a fresh clone of drupal_cms for testing with a fresh install and initialization of DDEV, on Apple silicon.
As specified in the description, here are the following desired metrics:
fubarhouse@drupal-cms-dev-web:/var/www/html$ composer show | grep drupal/ | wc -l Authenticity of packages from https://repo.packagist.org are not verified by TUF. 104
fubarhouse@drupal-cms-dev-web:/var/www/html$ du -sh vendor/composer/tuf 7.7M vendor/composer/tuf
I also ran into this one.... once - in a series of tests, however this seems inconsistent as I only got it once.
In TrustedAuthorityTrait.php line 50: [Tuf\Exception\MetadataException] The 'bin_c50-c53' contents does not match hash 'sha256' specified in the 'snapshot' metadata.
I have posted a couple of graphs in Drupal Slack if anybody is interested - they show the memory usage at ~800MB without the running process and at its peak it was using 7.1GB, however it did not run out of memory or show an error. A link to these graphs is below:
- 🇺🇸United States jimconte
We expect this to stabilize as we address the upstream issues. However, if you do encounter such an error, it is worthwhile to report it here, in this ticket.
I set up composer as described, and I'm running composer update -vvv manually.
I've been getting the same "Maximum allowed download size reached. Downloaded 1024 of allowed 1024 bytes" on these two for the past hour:
https://packages.drupal.org/8/metadata/bin_628-62b.json
https://packages.drupal.org/8/metadata/bin_04c-04f.jsonOn repeated runs, when one succeeds, the other fails.
- 🇺🇸United States drumm NY, US
Can you include a bit more context, ideally the full console output? What I was understanding in https://github.com/php-tuf/composer-integration/issues/128 was that the specific files weren’t mentioned in the error message. HTTP requests happen in parallel, so I don’t think the line above the error message is necessarily the cause, which would also explain why it appears to be switching.
- 🇺🇸United States jimconte
Piped output to log:
COMPOSER_MEMORY_LIMIT=-1 /usr/bin/php83 /usr/local/bin/composer28 -vvv update --no-ansi &> ../temp/composer.log
I am prototyping with a very basic install.
- 🇬🇧United Kingdom catch
I just tried this using the steps in the issue summary, and immediately got this error during composer update:
- Downloading drupal/core-project-message (11.x-dev 656efa0) Failed to download drupal/core-project-message from dist: Maximum allowed download size reached. Content-length header indicates 10666 bytes. Allowed 10658 bytes Now trying to download from source .... [ binary output ] .... In CurlDownloader.php line 374: [TypeError] rewind(): supplied resource is not a valid stream resource
Then I manually remove
drupal/core-project-message
from composer.json, and tried composer update again, got this:1/7 [====>-----------------------] 14% Failed to download drupal/core from dist: Maximum allowed download size reached. Content-length header indicates 26347482 bytes. Allowed 25427652 bytes Now trying to download from source
Are the steps in the issue summary outdated, or this is a regression somewhere?
- 🇬🇧United Kingdom catch
With a completely fresh install via composer create-project I was able to get the composer update to run. Maybe it's because I was on 11.x-dev?
Once I was able to run composer update, I did some profiling with xhprof, and opened https://github.com/php-tuf/php-tuf/issues/385 with initial findings.
- 🇬🇧United Kingdom catch
MR up at https://github.com/php-tuf/php-tuf/pull/386 which appears to be a 50% improvement for both CPU and memory.
- 🇺🇸United States drumm NY, US
- Downloading drupal/core-project-message (11.x-dev 656efa0) Failed to download drupal/core-project-message from dist: Maximum allowed download size reached. Content-length header indicates 10666 bytes. Allowed 10658 bytes
This is a server-side issue. https://packagist-signed.drupalcode.org/metadata/bin_094-097.json says
"drupal/core-project-message/11.9999999.9999999.9999999-dev": { "hashes": { "sha256": "78163df67a5207b9b0ba0ede7ff2652396f5c66ee9d6ce5f137db24d4242bd6e" }, "length": 10657 },
But https://packagist-signed.drupalcode.org/dist/drupal/core-project-message... is a different file. It was last updated Feb 3, which is around when I was shoring things up, so it's hard to say if it was a systematic issue or debris from something that was fixed. I'll look at getting that re-signed.
TUF is signing a target named
drupal/core-project-message/11.9999999.9999999.9999999-dev
and presumably it has a previous version of that file. For packagist-signed, dev zipballs do change location with every commit, the commit hash is in the filename. I wonder if there’s some way to use this for better error messaging or working around issues. - 🇬🇧United Kingdom catch
Second MR:
https://github.com/php-tuf/php-tuf/pull/387If I combine the two of these, memory usage in xhprof is more or less equivalent to vanilla composer.
- 🇺🇸United States drumm NY, US
- Downloading drupal/core-project-message (11.x-dev 656efa0) Failed to download drupal/core-project-message from dist: Maximum allowed download size reached. Content-length header indicates 10666 bytes. Allowed 10658 bytes
This is now fixed. I had put in logic that only signed each zip file once. The dev version zip archives have the Git commit ID in their name and change when a new commit is added. However, the target name
drupal/core-project-message/11.9999999.9999999.9999999-dev
isn’t changing and what is actually checked. The fix was https://gitlab.com/drupal-infrastructure/package-signing/packagist-signe... and https://gitlab.com/drupal-infrastructure/package-signing/packagist-signe... - 🇺🇸United States nicxvan
Ok I went through this with a mature Drupal project on 10.4.6
Highlights:
ddev composer show | grep drupal/ | wc -l
Authenticity of packages from https://asset-packagist.org are not verified by TUF.
Authenticity of packages from https://repo.packagist.org are not verified by TUF.
68du -sh vendor/composer/tuf
6.5M vendor/composer/tufHigh level impression, it took forever, it was almost 30 minutes I think for the update step and there was so much output it went past scrollback.
> pre-file-download: Tuf\ComposerIntegration\Plugin->preFileDownload
Steps taken:
ddev composer config allow-plugins.php-tuf/composer-integration true
No output
ddev composer require php-tuf/composer-integration:dev-main --dev
Abridged output:./composer.json has been updated Running composer update php-tuf/composer-integration Gathering patches for root package. Loading composer repositories with package information Updating dependencies Lock file operations: 4 installs, 0 updates, 0 removals - Locking paragonie/random_compat (v9.99.100) - Locking paragonie/sodium_compat (v1.21.1) - Locking php-tuf/composer-integration (dev-main 05a6d3e) - Locking php-tuf/php-tuf (0.1.6) Writing lock file
And
Authenticity of packages from https://packages.drupal.org/8 are not verified by TUF. Authenticity of packages from https://asset-packagist.org are not verified by TUF. Authenticity of packages from https://repo.packagist.org are not verified by TUF.
Next I ran the mkdir and curl commands, nothing remarkable
ddev composer tuf:protect https://packages.drupal.org/8 Authenticity of packages from https://packages.drupal.org/8 are not verified by TUF. Authenticity of packages from https://asset-packagist.org are not verified by TUF. Authenticity of packages from https://repo.packagist.org are not verified by TUF. https://packages.drupal.org/8 is now protected by TUF.
ddev composer config repositories.drupal-core composer https://packagist-signed.drupalcode.org Authenticity of packages from https://asset-packagist.org are not verified by TUF. Authenticity of packages from https://repo.packagist.org are not verified by TUF.
ddev composer tuf:protect https://packagist-signed.drupalcode.org Authenticity of packages from https://packagist-signed.drupalcode.org are not verified by TUF. Authenticity of packages from https://asset-packagist.org are not verified by TUF. Authenticity of packages from https://repo.packagist.org are not verified by TUF. https://packagist-signed.drupalcode.org is now protected by TUF.
- 🇺🇸United States trackleft2 Tucson, AZ 🇺🇸
Packages: 114
When I ran
composer -vv update
I sawPHP Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 4096 bytes) in /vendor/php-tuf/php-tuf/src/CanonicalJsonTrait.php on line 46 Check https://getcomposer.org/doc/articles/troubleshooting.md#memory-limit-errors for more info on how to handle out of memory errors.%
- 🇺🇸United States drumm NY, US
There have been significant memory usage improvements which need review linked from https://github.com/php-tuf/composer-integration/issues/127. Those could help the runtime as well.
- 🇬🇧United Kingdom catch
https://github.com/php-tuf/php-tuf/pull/395 just landed which was the last major memory improvement I found.
For me with just Drupal core, the combination of the three recent MRs brought memory usage down from about 320mb to 60mb. It would be great if people could test this again with some larger projects to see how it does when there are more packages to deal with.
- 🇪🇸Spain pcambra Asturies
I've tested this with a somehow large project I'm working on and after around 10 minutes or less I got:
Packages: 146
TUF metadata: 9.9M - 🇺🇸United States cmlara
This appears to needs an IS update to have those running the composer steps gather memory (and time if it really took 10 minutes) usage to be included in reports.
- 🇺🇸United States cmlara
Testing against Drupal CMS fails.
mkdir -p /tmp/dcms cd /tmp/dcms /usr/bin/time composer create-project drupal/cms cd cms /usr/bin/time composer -vv update composer config allow-plugins.php-tuf/composer-integration true composer require php-tuf/composer-integration:dev-main --dev mkdir tuf curl -o tuf/packages.drupal.org.json https://packages.drupal.org/8/metadata/1.root.json curl -o tuf/packagist-signed.drupalcode.org.json https://packagist-signed.drupalcode.org/metadata/1.root.json composer tuf:protect https://packages.drupal.org/8 composer config repositories.drupal-core composer https://packagist-signed.drupalcode.org composer tuf:protect https://packagist-signed.drupalcode.org composer -vv update
> pre-file-download: Tuf\ComposerIntegration\Plugin->preFileDownload In RootVerifier.php line 36: [Tuf\Exception\Attack\RollbackAttackException] Remote 'root' metadata version "$1" does not the expected version "$2" Exception trace: at /tmp/dcms/cms/vendor/php-tuf/php-tuf/src/Metadata/Verifier/RootVerifier.php:36 Tuf\Metadata\Verifier\RootVerifier->checkRollbackAttack() at /tmp/dcms/cms/vendor/php-tuf/php-tuf/src/Metadata/Verifier/RootVerifier.php:25 Tuf\Metadata\Verifier\RootVerifier->verify() at /tmp/dcms/cms/vendor/php-tuf/php-tuf/src/Metadata/Verifier/UniversalVerifier.php:59 Tuf\Metadata\Verifier\UniversalVerifier->verify() at /tmp/dcms/cms/vendor/php-tuf/php-tuf/src/Client/Updater.php:231 Tuf\Client\Updater->updateRoot() at /tmp/dcms/cms/vendor/php-tuf/php-tuf/src/Client/Updater.php:152 Tuf\Client\Updater->refresh() at /tmp/dcms/cms/vendor/php-tuf/composer-integration/src/ComposerCompatibleUpdater.php:51 Tuf\ComposerIntegration\ComposerCompatibleUpdater->getLength() at /tmp/dcms/cms/vendor/php-tuf/composer-integration/src/TufValidatedComposerRepository.php:220 Tuf\ComposerIntegration\TufValidatedComposerRepository->prepareComposerMetadata() at /tmp/dcms/cms/vendor/php-tuf/composer-integration/src/Plugin.php:55 Tuf\ComposerIntegration\Plugin->preFileDownload() at /var/www/html/vendor/composer/composer/src/Composer/EventDispatcher/EventDispatcher.php:220 Composer\EventDispatcher\EventDispatcher->doDispatch() at /var/www/html/vendor/composer/composer/src/Composer/EventDispatcher/EventDispatcher.php:115 Composer\EventDispatcher\EventDispatcher->dispatch() at /var/www/html/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:1653 Composer\Repository\ComposerRepository->asyncFetchFile() at /var/www/html/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:1111 Composer\Repository\ComposerRepository->startCachedAsyncDownload() at /var/www/html/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:1039 Composer\Repository\ComposerRepository->loadAsyncPackages() at /var/www/html/vendor/composer/composer/src/Composer/Repository/ComposerRepository.php:540 Composer\Repository\ComposerRepository->loadPackages() at /var/www/html/vendor/composer/composer/src/Composer/DependencyResolver/PoolBuilder.php:429 Composer\DependencyResolver\PoolBuilder->loadPackagesMarkedForLoading() at /var/www/html/vendor/composer/composer/src/Composer/DependencyResolver/PoolBuilder.php:279 Composer\DependencyResolver\PoolBuilder->buildPool() at /var/www/html/vendor/composer/composer/src/Composer/Repository/RepositorySet.php:332 Composer\Repository\RepositorySet->createPool() at /var/www/html/vendor/composer/composer/src/Composer/Installer.php:501 Composer\Installer->doUpdate() at /var/www/html/vendor/composer/composer/src/Composer/Installer.php:298 Composer\Installer->run() at /var/www/html/vendor/composer/composer/src/Composer/Command/UpdateCommand.php:281 Composer\Command\UpdateCommand->execute() at /var/www/html/vendor/symfony/console/Command/Command.php:279 Symfony\Component\Console\Command\Command->run() at /var/www/html/vendor/symfony/console/Application.php:1076 Symfony\Component\Console\Application->doRunCommand() at /var/www/html/vendor/symfony/console/Application.php:342 Symfony\Component\Console\Application->doRun() at /var/www/html/vendor/composer/composer/src/Composer/Console/Application.php:396 Composer\Console\Application->doRun() at /var/www/html/vendor/symfony/console/Application.php:193 Symfony\Component\Console\Application->run() at /var/www/html/vendor/composer/composer/src/Composer/Console/Application.php:136 Composer\Console\Application->run() at /var/www/html/vendor/composer/composer/bin/composer:99 include() at /var/www/html/vendor/bin/composer:119
- 🇬🇧United Kingdom catch
I tried the same steps as #39 and got the same result.
The error message made me wonder if there's a version 2 of the metadata, and there is https://packagist-signed.drupalcode.org/metadata/2.root.json
If I replace the following from the issue summary, after
rm -rf vendor/composer/tuf
then it works for me:curl -o tuf/packagist-signed.drupalcode.org.json https://packagist-signed.drupalcode.org/metadata/2.root.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2987 100 2987 0 0 1411 0 0:00:02 0:00:02 --:--:-- 1411 catch@*:~/www/cms$ curl -o tuf/packages.drupal.org.json https://packages.drupal.org/8/metadata/2.root.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2987 100 2987 0 0 2190 0 0:00:01 0:00:01 --:--:-- 2191
But this begs the question - if the new version is as a result of a key rotation or something, then how would a php-tuf enabled site be able to use new versions of the metadata?
- 🇺🇸United States drumm NY, US
We did recently refresh the root metadata ahead of the expiration later this month for both:
- https://packages.drupal.org/8/metadata/2.root.json
- https://packagist-signed.drupalcode.org/metadata/2.root.json
I would expect the client to fetch and use this automatically. If we refreshed and rotated keys in the root metadata as a result of a compromise, I would think the client should automatically start using the new metadata, which has a chain of trust back to 1.root.json. This should be checked against the TUF spec.
This will need some debugging to figure out where the root cause is.
- 🇯🇴Jordan Rajab Natshah Jordan
Should we have both in the same root
composer.json
file"drupal-core": { "type": "composer", "url": "https://packagist-signed.drupalcode.org", "tuf": true }, "drupal": { "type": "composer", "url": "https://packages.drupal.org/8", "tuf": true },
Or only one
"drupal-core": { "type": "composer", "url": "https://packagist-signed.drupalcode.org", "tuf": true },
I keep hitting
https://packages.drupal.org/8 could not be fully loaded (Maximum allowed download size reached. Content-length header indicates 7240 bytes. Allowed 1024 bytes), package information was loaded from the local cache and may be out of date
And I see this
https://packagist-signed.drupalcode.org/metadata/3.root.json
too. - 🇺🇸United States drumm NY, US
https://packagist-signed.drupalcode.org/metadata/3.root.json
is by design, that will only exist when the root metadata is refreshed for a 3rd time.I keep hitting https://packages.drupal.org/8 could not be fully loaded (Maximum allowed download size reached. Content-length header indicates 7240 bytes. Allowed 1024 bytes), package information was loaded from the local cache and may be out of date
Until https://github.com/php-tuf/composer-integration/issues/128 is resolved, I have basically no capability to help with issues like this. I need to know which HTTP request is causing the problem.
- 🇺🇸United States drumm NY, US
Both https://packagist-signed.drupalcode.org and https://packages.drupal.org/8 should be in composer.json with
"tuf": true
https://packages.drupal.org/8 only includes contrib modules and themes from Drupal.org. The rest of Drupal, including core, is distributed via Packagist.org, which does not have code signing. packagist-signed.drupalcode.org mirrors packagist.org, so core and the TUF library are signed.
- 🇯🇴Jordan Rajab Natshah Jordan
Got it
Thanks, Neil, for the tips and notes.
Doing one day round of testing with TUF, Auto-update for a real case. ( figuring out the robot workflow for auto updating a live site - with a cloned copy in a stage host.
Interesting finding and stepsThinking of crating two DDEV commands ( or bash ) like
ddev init-autoupdate ddev disable-autoupdate
As I think a cloned local or staged copy could work with DDEV or Docker directly
1- Clone
2- Init Autoupdate
3- Auto update
4- Automated Unit/Functional testing process ( Get a pass flag )
5- Disable autoupdate
6- Git commit
7- DeployThank you so much for working on TUF and the Package Manager for Drupal!!
- 🇺🇸United States cmlara
Until https://github.com/php-tuf/composer-integration/issues/128 is resolved, I have basically no capability to help with issues like this.
Given we had issues in #30 is there any background process running to routinely verify the integrity of the repository (and alert when integrity is breached) that could identify these issues prior to a report from a user and without need to resolve the issue in TUF?
- 🇺🇸United States drumm NY, US
We are monitoring for known causes of integrity issues, such as queues not being processed promptly. Verifying the entire repository would probably take multiple days and have little practical benefit for spotting issues which I suspect are only present for a few minutes at a time.