[meta] Deprecate tarballs, because they are incompatible with Composer-managed dependencies, Automatic Updates, Project Browser, and release managers' health

As a start for #5. The wiki page is updated with a warning that tarballs are being deprecated.

What about combining 1 and 2 and move both

Tarball downloads are currently featured prominently on https://www.drupal.org/download →

Made issue for Step 6.

Also the evaluator guide → needs to rewritten since it uses the tar-ball for quick setup.

@hansfn, Thanks!

I've update the IS with a new step in the remaining tasks where links can be added to any other pages that needed to be changed.

Maybe I'm missing something, but composer doesn't magically remove the need for tarballs. It's just a different (mostly better) way to install tarballs for you. But that's what a "dist" installation is. Composer has to get your code from somewhere. Either it's "prefer source" (Git checkout) or "prefer dist" (a "tarball").

1. Yes, we can hide them from the UI of release nodes. 📌 Remove download of tar files Postponed Not sure why that's postponed, seems it could proceed any time.
2. Yes, we can deprecate the parts of Update Manager that let you install "directly from tarball" via the UI once we have project browser. I believe that's 📌 Remove adding an extension via a URL Fixed
3. Yes, we can deprecate the parts of Update Manager that update your currently installed codebase once we have autoupdates. Is that #3201968: Augment then Replace current Update Manager URL download based updates with Staged-Composer workflow → ? The summary here lists #3417136 twice. 3201968 has its own ancestry via autoupdates, not sure we should re-parent it.

But none of that means we can "Deprecate tarballs", because all of it still depends on a dist release artifact under the hood. E.g. if you run composer require drupal/slim_select, you'll eventually see:

  - Installing drupal/slim_select (1.0.0-alpha4): Extracting archive

"Extracting archive" means tar -zxf slim_select-1.0.0-alpha4.tar.gz in this case. 😅

We can't actually do #3417136 or #3201968 (or whatever issue we should use for point 2) until we have a stable replacement. #3419966 could happen anytime.

@dww this is only about the core packaged tarballs like https://ftp.drupal.org/files/projects/drupal-10.2.3.tar.gz that include /vendor code and are generated by the packaging script. drupal/core, drupal/core-recommended etc. would still be in .tar.gz format but they're only pulled via composer and not exposed for direct download and install. Updated the title to make this a bit more explicit.

This in the issue summary hasn't been true

Tarball installations are completely incompatible with modules that declare their own Composer dependencies. Furthermore, switching from a tarball installation to a Composer installation on an existing site is a difficult process for many users.

The tarballs are made with composer create-project legacy-project.

It is somewhat good to have tarballs as a test in packaging, to make sure composer create-project works.

@drumm I think that statement means: if you download core and Commerce as tarballs, Commerce won't work because it has its own Composer dependencies that never get merged into the top-level vendor directory. And if you do have a working tarball setup, to switch to Composer you need to manually add each of your dependencies to composer.json, there is no automatic way of doing this.

We do have build tests in core to ensure that create-project works, but obviously they can't test a released version until the version is actually released :)

➜  ~/Downloads tar xf tar.gz
➜  ~/Downloads cd drupal-10.2.5
➜  ~/Downloads/drupal-10.2.5 composer require 'drupal/commerce:^2.38'
./composer.json has been updated
Running composer update drupal/commerce
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/commerce ^2.38 -> satisfiable by drupal/commerce[2.38.0].
    - drupal/commerce 2.38.0 requires drupal/inline_entity_form ^1.0@RC || ^3.0@RC -> found drupal/inline_entity_form[dev-1.x, dev-3.x, 1.0.0-alpha1, ..., 1.x-dev (alias of dev-1.x), 3.0.0-rc15, ..., 3.x-dev (alias of dev-3.x)] but it does not match your minimum-stability.


Installation failed, reverting ./composer.json and ./composer.lock to their original content.
➜  ~/Downloads/drupal-10.2.5 composer require drupal/inline_entity_form ^1.0@RC
./composer.json has been updated
Running composer update drupal/inline_entity_form
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking drupal/inline_entity_form (1.0.0-rc17)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Downloading drupal/inline_entity_form (1.0.0-rc17)
  - Installing drupal/inline_entity_form (1.0.0-rc17): Extracting archive
Generating autoload files
Hardening vendor directory with .htaccess and web.config files.
42 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Cleaning installed packages.
No security vulnerability advisories found.
➜  ~/Downloads/drupal-10.2.5 composer require 'drupal/commerce:^2.38'
./composer.json has been updated
Running composer update drupal/commerce
Loading composer repositories with package information
Updating dependencies
Lock file operations: 10 installs, 0 updates, 0 removals
  - Locking commerceguys/addressing (v2.2.0)
  - Locking commerceguys/intl (v2.0.5)
  - Locking doctrine/collections (2.2.1)
  - Locking drupal/address (2.0.1)
  - Locking drupal/commerce (2.38.0)
  - Locking drupal/entity (1.4.0)
  - Locking drupal/entity_reference_revisions (1.11.0)
  - Locking drupal/profile (1.10.0)
  - Locking drupal/state_machine (1.11.0)
  - Locking drupal/token (1.13.0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 10 installs, 0 updates, 0 removals
  - Downloading doctrine/collections (2.2.1)
  - Downloading commerceguys/addressing (v2.2.0)
  - Downloading drupal/token (1.13.0)
  - Downloading drupal/state_machine (1.11.0)
  - Downloading drupal/entity (1.4.0)
  - Downloading drupal/profile (1.10.0)
  - Downloading drupal/address (2.0.1)
  - Downloading commerceguys/intl (v2.0.5)
  - Downloading drupal/commerce (2.38.0)
 0/9 [>---------------------------]   0%
 5/9 [===============>------------]  55%
 6/9 [==================>---------]  66%
 8/9 [========================>---]  88%
 9/9 [============================] 100%
  - Installing doctrine/collections (2.2.1): Extracting archive
  - Installing commerceguys/addressing (v2.2.0): Extracting archive
  - Installing drupal/token (1.13.0): Extracting archive
  - Installing drupal/state_machine (1.11.0): Extracting archive
  - Installing drupal/entity (1.4.0): Extracting archive
  - Installing drupal/profile (1.10.0): Extracting archive
  - Installing drupal/entity_reference_revisions (1.11.0): Extracting archive
  - Installing drupal/address (2.0.1): Extracting archive
  - Installing commerceguys/intl (v2.0.5): Extracting archive
  - Installing drupal/commerce (2.38.0): Extracting archive
  0/10 [>---------------------------]   0%
  7/10 [===================>--------]  70%
  9/10 [=========================>--]  90%
 10/10 [============================] 100%
Generating autoload files
Hardening vendor directory with .htaccess and web.config files.
43 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Cleaning installed packages.
No security vulnerability advisories found.

Sorry, I missed this part:

and Commerce as tarballs

Tarballs are indeed not good for contrib, and https://www.drupal.org/project/commerce/releases/8.x-2.38 → says to use Composer over downloads.

Core tarballs are composer-compatible, you get the same thing as composer create-project drupal/legacy-project

I'm not sure why you would do #26 - if you have Composer then you can run create-project yourself already? But given that example maybe core tarballs aren't really a problem, it's contrib ones that can be problematic. Maybe we can start on the remaining tasks like removing the Updater UI, and start removing more references to tarballs from d.o without waiting for Automatic Updates?

Is there any way of telling from access logs how many people still download tarballs manually compared to via Composer? Unsure also how we will achieve point 3 of notifying existing tarball users that they should switch to Composer.

Yes, we should go ahead and continue discouraging using contrib tarballs directly. They are still totally used by Composer internally of course.

We prominently feature core tarballs on https://www.drupal.org/download → and various other places mentioned in this issue. Those need auditing & updating before we think about removing core tarballs. Otherwise, I'm mostly mentioning that core tarballs do work with composer, and the issue summary needs updating.

Is there any way of telling from access logs how many people still download tarballs manually compared to via Composer?

We could probably get the access logs from S3, filter to tar/zip release downloads, and see how many have Composer as a user agent. That would be good numbers for contrib, since Composer and manual downloads are the same URL. We would not get a comparison for core, since the downloads via Composer don’t hit our servers, we’d only see the volume of non-composer downloads.

Unsure also how we will achieve point 3 of notifying existing tarball users that they should switch to Composer.

Could core check for the codebase mismatching composer's installed.json? I think automatic updates already has a check for that.

Can an alternative installer be provided instead that simply (potentially bundles and) runs a composer command to download and install drupal?

Would it help to remove this before D7 EOL, if we still need to keep all the code for 7.x tarballs?

I do not know the usage stats, but if these are relatively high, would it make sense to remove these tarballs before Project Browser / Auto Updates?

D7 core actually has the same packaging as contrib. Only core D8+ is special, and will remain special for subtree splits, which are part of packaging.

What we do get to simplify around D7 EOL is special cases for displaying tarballs. At some point https://www.drupal.org/project/webform/releases/7.x-3.31 → could look the same as https://www.drupal.org/project/webform/releases/6.2.2 → , probably not for a few months after D7 EOL since there will inevitably be stragglers. And D7 packages.drupal.org support will have to remain until usage is sufficiently low.

Tried to add clarification to the first item in the issue summary. Rearranged the remaining task to be more like the order in which the items can be done.

Do we have a date when sites using tarballs will no longer be available and site must switch to composer?

@quietone

Do we have a date when sites using tarballs will no longer be available and site must switch to composer?

No as I understand it, this issue is for deciding when that date would be amongst other details of deprecating tarballs.

What I think might make sense is to pick a date that is a few months in the future and announce 'Drupal core tarball installs will no longer be supported after x date' so people have plenty of warning, and then we can figure out all the steps between now and then. It wouldn't be a deadline, but it would mean we don't end up stuck here because we didn't do an announcement in time.

Given we have a convenient future date of 5 January 2025 for Drupal 7 EOL should we announce the same date for EOL of tarballs as well?

That works for me - even if d.o doesn't actually drop support for a few months afterwards it still makes sense to announce as the policy change.

I agree with setting the date to 5.1.2025 to match the D7 EOL.

📌 Remove adding an extension via a URL Fixed was listed for both point 3 and 4. I replaced the mention in point 3 with #3201968: Augment then Replace current Update Manager URL download based updates with Staged-Composer workflow → which I think is closer. Remember, the current "Update Manager" has 2 main aspects:

- Update your currently installed stuff (point 3) -> to be replaced with Auto Updates
- Install new stuff (point 4) -> to be replaced with Project Browser

They really need to be handled separately, so let's not conflate them and use the same issue to track both.

Meanwhile, I still think point #6 (the d.o UI) could happen any time, I don't think that needs to be blocked on everything else.

All that said, I think we need to be careful how we word this effort and the related communications. As I wrote at #22, we're still going to have tarballs, even "packaged" ones. We're just telling people they have to use composer to install them. Maybe I'm hung up on pedantic semantics, but I believe the title here is misleading and confusing.

Updated the Issue Summary and title to reflect the recent comments.

As for the wording, should we start a google doc for that?

For #6 It doesn't seem right to remove the ability to download until the deprecation date. In the meantime a warning could be added.

Maybe this for the re-title?

Thanks for addressing my concerns. I believe @quietone's version was closer to reality. 😅 This issue started as (paraphrase) "Release managers hate tarballs of core that include vendor", but the title from #41 is going in the wrong direction in a few ways:

1. The Update Manager doesn't do anything with core, tarballs or otherwise. 😂
2. There are many other things to deal with if we want to stop letting folks directly install from tarball than just Update Manager (d.o changes, communications, etc). These are spelled out in the summary, but the title from #41 only covers 2 of 6 points.

Therefore, restoring the title from #40 (until something better comes along).

Thanks again,
-Derek

whoops, just notice the grammar bug, sorry for the noise

p.s. I just opened 📌 RFC for Ludwig maintainers: Core wants to require composer for everything Active to alert the Ludwig maintainers about this issue, since I figured those folks might care and want to chime in...

Arggh I thought I was on a different issue entirely.

I thought this issue was originally about Drupal core. Remove tar/zip download links from https://www.drupal.org/download → , https://www.drupal.org/project/drupal/releases/10.2.5 → , and elsewhere, focusing on only composer for installing Drupa. Then dismantle as much of the supporting infrastructure as we want.

Some of the reasons in the issue summary are not true, and others are new, like 📌 Remove drupal/legacy-project as a starting point for Drupal 10 Postponed .

For contrib, packaging and tarballs remain, download links may be more-hidden or removed from release pages, and 📌 Remove adding an extension via a URL Fixed moves forward.

A possibly better title might be "Only support Core installs managed by composer" so that we get away from the argument about the existence of tarballs in the backed.

After retitling this and reading #46 it seems to me that this is not dependent on Auto Updates and Project Browser (as I have though previously), as we are talking mostly about core installs. So if I understand it correctly, after we remove these download links, it would not be possible to install Drupal for anyone without the knowledge of composer (if they do not know about the github repository)? So all users currently doing download - extract - install will be cut out, if they do not understand composer? Then I think it would be great to have a look at the stats mentioned in #28/#29. Because this change could increase the level of knowledge for needed for even installing the basic Drupal installation for newcommers (mostly for those without deeper technical knowledge). It is good to take this factor into account as well.

Considering sites will have already designed and implemented their D10 deployment processes, should this be reserved for a major release onward so that site owners can implement this at the same time they typically plan to refactor their deployments? Possibly change this to "In the release of D11 onward"

Positive:

• Moves the first implementation of this forward
• Allows sites a more delineated changeover that they can choose to adopt when ready (either in the D10 release cycle before upgrading to D11, or when they choose to upgrade to D11)
• Helps encourage sites upgrading from D7 to implement the the new workflow.

Negative:
D10 is now slated to survive until D12 release in 2026 meaning full removal would be considerably longer possibly decreasing the overall gain to Release Managers.

Re Statsticis:
Would those be biased towards composer due to CI runs? Since they do not have core releases do they tell us much about the actual ecosystem?

We’re plowing ahead with at least removing the UI to install new stuff. 📌 Remove adding an extension via a URL Fixed now has 2 MRs:

• Full removal for 11.x branch: https://git.drupalcode.org/project/drupal/-/merge_requests/8781 (removes routes, links, forms, dead code, adds deprecations, etc).
• Backport version for 11.0.x (and 10.*.x branches?) https://git.drupalcode.org/project/drupal/-/merge_requests/8810 (only removes routes, action links, and tests, leaves all the dead code in place).

I’m not interested in moving all this now broken stuff to a contrib module and deprecating it. It served a need for a long time, and I’m glad I wrote it when I did, but it should go, even without project browser in core.

I’m a little more torn about the updater parts. I think we really need a working replacement before we remove it all. But perhaps we could at least formally deprecate all that (not exactly sure how we deprecate a form and route, honestly).

P.s. as Drumm has pointed out multiple times, parts of the summary aren’t true.

Latest composer 2.8.1 became stricter on tags so it may affect split https://github.com/composer/composer/issues/12150#issuecomment-2423790429

📌 Deprecate/remove the ability to update a module from a URL and authorize.php Active is now the correct dedicated issue for deprecating / removing the authorize.php code and the UI from update.module.