Manually test TUF-enabled Composer projects

Tested on a small personal project. Results:

• Packages: 15
• TUF metadata: 2.9M

Tested on a second small side-project:

Packages: 19
TUF metadata: 2.8M

I'm planning to test on a bunch of different projects. To make this more efficient, I put together a couple simple scripts that just run the steps outlined in the issue summary.

Report from a medium-sized application:

Packages: 53
TUF metadata: 3.6M

When running this composer tuf:protect https://packagist-signed.drupalcode.org I got the following:

Authenticity of packages from https://packagist-signed.drupalcode.org are not verified by TUF.

[red] There are no commands defined in the "tuf" namespace. [/red]

Composer [tuf:protect https://packagist-signed.drupalcode.org] failed, composer command failed: exit status 1. stderr=

I edited the composer.json file manually to have:

        "drupal-core": {
            "type": "composer",
            "url": "https://packagist-signed.drupalcode.org",
            "tuf": true
        },

And then I could continue.

Report:
- Packages: 32
- TUF metadata: 3.0M

I re-did the whole thing from scratch on the above project and I did not get that small error, so all the steps worked directly.

FYI, you may occasionally see an error like:

The 'bin_c50-c53' contents does not match hash 'sha256' specified in the 'snapshot' metadata.

Sometimes this is transitory, and will fix itself in a matter of seconds. Re-running `composer update` will likely succeed. If not, but the 'bin_c50-c53' part changes, wait a minute or two and try again. This is a symptom of temporary inconsistency during metadata updates on the server-side. This is being addressed upstream in rugged/rugged#208: Ensure metadata consistency.

On the other hand, this kind of error can also persist longer. If re-running `composer update` show no change in the 'bin_c50-c53' part of the error message, this likely indicates that the TUF server has "stalled" during a metadata update. This will generally require intervention by a TUF server administrator. This is being addressed upstream in rugged/rugged#204: Add a monitor task to clean up stuck targets.

We expect this to stabilize as we address the upstream issues. However, if you do encounter such an error, it is worthwhile to report it here, in this ticket.

I did the process and it didn't work initially. As @fjgarlin said in #8, only after I did the process from start a second time it did work. Maybe we should have first executed the code to download the initial root metadata for the repos and only then try to add the signed repository for Drupal core and enable TUF protection for it? I mean, it doesn't seem like an issue of the Packaging component but more like an incorrect order of the commands as described in the issue summary here.

Packages: 77
TUF metadata: 4.5M

I also had problems with installation and now get out of memory errors when running composer up in both ddev and on my MacOS host.
PHP Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/php-tuf/php-tuf/src/CanonicalJsonTrait.php on line 46

FWIW, here is my data on one client's installation after several attempts:
Packages: 118
TUF metadata: 5.3M

Hit an immediate hard failure - opened PR https://github.com/php-tuf/composer-integration/pull/126
With applying that, get:

https://packagist-signed.drupalcode.org could not be fully loaded (Maximum allowed download size reached. Downloaded 1165080 of allowed 1165080 bytes), package information was loaded from the local cache and may be out of date

(this continues after many runs)

Packages: 81
TUF metadata: 5.4mb

Failing with "The 'bin_d9c-d9f' contents does not match hash 'sha256' specified in the 'snapshot' metadata." - for a while.

I haven't done the steps yet but #10 suggests this needs work one way or the other.

On the other hand, this kind of error can also persist longer. If re-running `composer update` show no change in the 'bin_c50-c53' part of the error message, this likely indicates that the TUF server has "stalled" during a metadata update. This will generally require intervention by a TUF server administrator. This is being addressed upstream in rugged/rugged#204: Add a monitor task to clean up stuck targets.

https://gitlab.com/drupal-infrastructure/package-signing/rugged-metrics now alerts us when targets are stuck, but it does require manual intervention to resolve. Once https://gitlab.com/rugged/rugged/-/issues/204 is deployed, this will be more automatic.

Most of the issues we have seen are actually directories stuck in the building phase, before Rugged is involved. There is also some chance of disruption on Tuesdays during the regular AWS maintenance window for the managed queue.

PHP Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/php-tuf/php-tuf/src/CanonicalJsonTrait.php on line 46

1.6MB is not a lot, especially when Drupal will require 64MB. Regardless, the php-tuf client will need to be better at detecting constrained memory situations, and keeping memory usage low. We do want to be judicious with memory use, since the hope is to expand TUF signing to the wider PHP ecosystem.

https://packagist-signed.drupalcode.org could not be fully loaded (Maximum allowed download size reached. Downloaded 1165080 of allowed 1165080 bytes), package information was loaded from the local cache and may be out of date

I think the Composer plugin client will need some error handling improved. This suggests something is amiss server-side, good chance cached at the CDN, but I don’t know what it might be. Running with -vvv might help point to which URL might be the issue.

Failing with "The 'bin_d9c-d9f' contents does not match hash 'sha256' specified in the 'snapshot' metadata." - for a while.

This looks like it is resolved now:

$ curl --silent https://packagist-signed.drupalcode.org/metadata/snapshot.json | grep -A2 bin_d9c-d9f
   "bin_d9c-d9f.json": {
    "hashes": {
     "sha256": "bf9500067989a47152416e9dde6b7c059286657d080f2fd526949e462d5c0172",
$ curl --silent https://packagist-signed.drupalcode.org/metadata/bin_d9c-d9f.json | sha256sum
bf9500067989a47152416e9dde6b7c059286657d080f2fd526949e462d5c0172  -

Right now we are permanently caching at the CDN and purging when changes happen. I’ve seen caching be enough of an issue that I think I will put a 5 or 10 minute lifetime on the cache at the CDN. So issues like this should self-heal after a few minutes.

Opened a few issues to track what I mentioned in my last comment:

• Improve resilience of uploading building directories https://gitlab.com/drupal-infrastructure/package-signing/packagist-signe... this is an operational improvement. When a building directory is stuck, rugged continues as usual
• Improve handling of low-memory situations https://github.com/php-tuf/composer-integration/issues/127
• Improve error messages to show which HTTP request is the source of errors https://github.com/php-tuf/composer-integration/issues/128
• The fallback TTL of 1 hour was being used for TUF metadata at the CDN. It is now reduced to 10 minutes

I moved “To provide the root of trust for TUF verifications, download the initial root metadata for these two repos:” up in the issue summary, which should make the setup steps run more smoothly. While there are things to improve, we do need more real-world testing to see if the improvements have been effective, so setting back to needs review.

I also got the same result as #7 and @fjgarlin's notes fixed it.
I had to increase my php memory limit to run composer -vv update

Report from my medium D11 site:

- Packages: 31
- TUF metadata: 3.9M

Same here as the previous tests, some memory issues, plus the order of commands.
My results:

- Packages: 89
- TUF metadata: 6.6M

Putting my edits to address #7 back in the issue summary.

skyejohnson & kyriazo - what was your PHP memory limit that you hit when you started, and how much did you raise it to succeed?

@drumm I just set it to -1 tbh

Using the time utility (make sure its the binary one not the bash built-in) would also probably be useful to log memory usage.

I put this into a contrib module lab (in no way representative of an actual deployment, just 10 packages of drupal/core using 3.0 mb in the tuf folder to develop a single contrib module)

With the tuf plugin:

$ /usr/bin//time composer -vv update
22.15user 3.12system 0:50.02elapsed 50%CPU (0avgtext+0avgdata 975440maxresident)k
88inputs+297600outputs (0major+270631minor)pagefaults 0swaps

Without the tuff plugin:

$ /usr/bin//time composer -vv update
10.19user 3.09system 0:20.90elapsed 63%CPU (0avgtext+0avgdata 280376maxresident)k
0inputs+285856outputs (0major+90243minor)pagefaults 0swaps

Max resident memory consumption increase, 695mb, a 3.4x increase..
(Jut to be clear #14 mentions 1.6Mb, the OOM report is actually 1.6Gb)

COMPOSER_MEMORY_LIMIT=960m - succeed with tuff (succeeds by less than 64mb)
COMPOSER_MEMORY_LIMIT=256m - succeed without tuf (by less than 10mb)

I misread the conversion here:

PHP Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/php-tuf/php-tuf/src/CanonicalJsonTrait.php on line 46

1.6MB is not a lot, especially when Drupal will require 64MB. Regardless, the php-tuf client will need to be better at detecting constrained memory situations, and keeping memory usage low. We do want to be judicious with memory use, since the hope is to expand TUF signing to the wider PHP ecosystem.

This is actually 1.6GB, which is already quite a high memory limit. Along with #21, it does look like the client seems to have a memory usage problem. So https://github.com/php-tuf/composer-integration/issues/127 is looking like the biggest blocker to getting this closer to core.

Neil brought this to my attention, so I've been taking a quick look at this as part of DrupalCon contribution day in Singapore.

I've used a fresh clone of drupal_cms for testing with a fresh install and initialization of DDEV, on Apple silicon.

As specified in the description, here are the following desired metrics:

fubarhouse@drupal-cms-dev-web:/var/www/html$ composer show | grep drupal/ | wc -l
Authenticity of packages from https://repo.packagist.org are not verified by TUF.
104

fubarhouse@drupal-cms-dev-web:/var/www/html$ du -sh vendor/composer/tuf
7.7M	vendor/composer/tuf

I also ran into this one.... once - in a series of tests, however this seems inconsistent as I only got it once.

In TrustedAuthorityTrait.php line 50:

  [Tuf\Exception\MetadataException]
  The 'bin_c50-c53' contents does not match hash 'sha256' specified in the 'snapshot' metadata.

I have posted a couple of graphs in Drupal Slack if anybody is interested - they show the memory usage at ~800MB without the running process and at its peak it was using 7.1GB, however it did not run out of memory or show an error. A link to these graphs is below:

• https://drupal.slack.com/archives/C7QJNEY3E/p1733886574903149?thread_ts=...