Decide on & implement targets management for Rugged instance

Created on 21 March 2023, over 1 year ago
Updated 22 July 2024, 3 months ago

Rugged, implementing the TUF spec, keeps a copy of everything signed, the targets. For Drupal.org, these are Composer metadata on packages.drupal.org, and zip file downloads on ftp.drupal.org. Hosting for these is remaining as-is and we are not making the TUF targets publicly accessible.

It appears that the targets can be safely removed once processing is complete by Rugged. To save disk, we could look at not keeping the targets directory. Potential solutions:

  • Not have rugged save targets
  • Empty targets on cron
  • Or set up some filesystem trick so files placed in targets go nowhere

The next steps are:

  • Determine if this is actually a problem, it might be okay to keep an extra copy of everything - it would be
  • Determine if Rugged needs the targets to remain in-place while processing; we don’t want them to disappear when needed
📌 Task
Status

Fixed

Component

Updates System

Created by

🇺🇸United States drumm NY, US

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @drumm
  • 🇺🇸United States drumm NY, US

    To get an idea of what order of magnitude we're talking about - 650G is every packaged release of a Drupal project.

    We’re only signing the zip files, since that’s what Composer uses. Zip files tend to be slightly larger than tgz, so let’s call it 400G.

    We are only signing “current” releases, compatible with Drupal 8 and later. Let’s ignore that, since it’s not bad to over-estimate, and we have an addition next.

    We are also signing composer metadata, which is 2 json files like https://packages.drupal.org/files/packages/7/p2/drupal/ctools.json per project.

    So the space needed for targets is probably less than half a terabyte, for now.

  • 🇺🇸United States drumm NY, US

    Given that amount space used, we do want to solve this. Even if we eat the cost of the extra storage, operations like backup and restore, and other maintenance would take orders of magnitude longer.

    A good, straightforward option is for something to delete files older than an hour on cron.

  • 🇨🇦Canada ergonlogic Montréal, Québec 🇨🇦

    I expect that we're going to want to implement consistent snapshots at some point. See: § 6.2.1 Writing consistent snapshots for specifics. In that case, Rugged MUST keep multiple copies of each target file named: <HASH>.FILENAME.EXT. This would obviously increase the storage requirements significantly.

    However, at the moment, I don't think that Rugged needs to keep processed targets. So we could have Rugged remove the target files.

  • 🇨🇦Canada ergonlogic Montréal, Québec 🇨🇦

    I added the following feature request w/ Rugged: #132: Add option to delete target files once they're signed

  • 🇨🇦Canada ergonlogic Montréal, Québec 🇨🇦
  • Status changed to Fixed 3 months ago
  • 🇺🇸United States drumm NY, US

    I believe we have this resolved.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024