General projects on Packagist.org have empty dist information

If I run

composer require -vvv 'drupal/inline_entity_form:^3.0@RC'

the zip file is used

  - Downloading drupal/inline_entity_form (3.0.0-rc20)
Downloading https://ftp.drupal.org/files/projects/inline_entity_form-3.0.0-rc20.zip
[200] https://ftp.drupal.org/files/projects/inline_entity_form-3.0.0-rc20.zip
Writing /Users/drumm/Library/Caches/composer/files/drupal/inline_entity_form/bdb2c802601e9865867bc13142b06aaccaab898d.zip into cache from /Users/drumm/Downloads/install-dir/vendor/composer/tmp-be7614412223e13c450c6ff1e182ba8f.zip

And composer show with a version specified also shows the dist file:

composer show drupal/inline_entity_form '^3.0@RC'
name     : drupal/inline_entity_form
descrip. : Provides a widget for inline management (creation, modification, removal) of referenced entities.
keywords :
versions : * 3.0.0-rc20
type     : drupal-module
license  : GNU General Public License v2.0 or later (GPL-2.0-or-later) (OSI approved) https://spdx.org/licenses/GPL-2.0-or-later.html#licenseText
homepage : https://www.drupal.org/project/inline_entity_form
source   : [git] https://git.drupalcode.org/project/inline_entity_form.git 3.0.0-rc20
dist     : [zip] https://ftp.drupal.org/files/projects/inline_entity_form-3.0.0-rc20.zip 3.0.0-rc20

I think your command is showing that the dev release does not have a dist file, which is true for packages.drupal.org.

If there's no git this does not work. For example if you do in some environment that has docker:

docker run --name no_git ddev/ddev-webserver:v1.23.3 &
docker exec -it no_git /bin/bash

Then run the commands on the container:

rm /usr/bin/git
composer create-project drupal/recommended-project
cd recommended-project/
composer require -vvv 'drupal/inline_entity_form:^3.0@RC'

You will see:

In GitDownloader.php line 82:

  [RuntimeException]
  git was not found in your PATH, skipping source download

This is unexpected. If you fix up the composer.json and then try to require a stable module like pathauto it works as expected.

Ah, this has nothing to do with RC releases, it is the dependency on drupal/rat, which is a general project with metadata hosted by Packagist.org.

  - Installing drupal/rat (1.0.0): Cloning 28202b02262a39ac8dbbfd43696b67c0c8c46b71 from cache
Executing command (CWD): git clone --no-checkout '/Users/drumm/Library/Caches/composer/vcs/https---git.drupalcode.org-project-rat.git/' '/Users/drumm/Downloads/install-dir/vendor/drupal/rat' --dissociate --reference '/Users/drumm/Library/Caches/composer/vcs/https---git.drupalcode.org-project-rat.git/' && cd '/Users/drumm/Downloads/install-dir/vendor/drupal/rat' && git remote set-url origin -- 'https://git.drupalcode.org/project/rat.git' && git remote add composer -- 'https://git.drupalcode.org/project/rat.git'

We currently do have the archive functionality of GitLab walled off, there is some explanation in the related issues I’m adding. We can allow archive downloads and see if Packagist/Composer will picks those up for general projects.

We now allow archive downloads from git.drupalcode.org. Packagist.org does not seem to have picked this up.

• I triggered a Packagist update for drupal/rat and https://packagist.org/p2/drupal/rat.json still has "dist" : null
• I added drupal/general_example to Packagist for the first time, https://packagist.org/p2/drupal/general_example.json also has "dist" : null

A package hosted on GitLab.com via Packagist should be checked to see if it has dist populated, to see if there is general GitLab support here. If there is, likely something needs to be updated to tell Packagist.org that git.drupalcode.org is also GitLab, and has a working archive API.

https://packagist.org/p2/open-source-developer/reaktr-core.json does have dist information. I'll contact the Packagist.org maintainers to see if there is something we should do to complete this.

https://github.com/composer/packagist/issues/1464 is the issue with Packagist.org to track this.

Packagist.org does not currently support this: https://github.com/composer/packagist/issues/1464

How important is this practically? Is this a blocker for a major initiative? Does anyone want to help with the Packagist issue?

From a Drupal.org perspective, this is works as designed.

@drumm thanks for your efforts here. Does the answer on github mean that we should add the gitlab-domains config to our composer.json and advise existing projects to do that too?

I gave composer.json changes a couple tries at

https://git.drupalcode.org/project/general_example/-/commit/c80581416079...
https://git.drupalcode.org/project/general_example/-/commit/2980bde1c969...

Neither made any difference. I also spent some time looking through packagist & composer source code to see what it might be doing. I don’t have anything conclusive, but I did see some hard-coding gitlab.com and what looked like type being vcs in packagist.

packagist.org is indeed not aware of self-hosted gitlab instances, as supporting them requires configuring the gitlab-domains setting in the composer config. Composer does not auto-detect that a domain is hosting a gitlab instance.

I’m thinking that likely means composer config for packagist itself, not any individual package. I totally understand Packagist.org not wanting to manually maintain a list of self-hosted GitLabs, and various side effects & edge cases that might follow.

Found this thanks to alex. I'm here to share the results of some testing based on the conversation.

I added the domain to my composer.json:
"gitlab-domains": "git.drupalcode.org"

composer show -a drupal/rat --no-cache still does not show a dist, and in the case of my work computer behind a proxy, it also only shows the -dev version (it is type: library, not drupal-module which I think is related to the issue).

I then added a personal access token to let it try and get the unstable/dev version, and that failed with 403.

Note that is affects DTT as well - https://git.drupalcode.org/project/dtt/. We just moved to drupal.org for project hosting and it would be nice to have dist downloads instead of git clone.

Would it make sense to try to serve these projects from the packages.drupal.org endpoint, and not rely on packagist? Or maybe some other way to avoid being recognized as a Gitlab repo type?

Most followups should be at the underlying issue: https://github.com/composer/packagist/issues/1464

This actually could become a priority higher than “nice to have” for automatic updates. Automatic updates is designed to sign dist files, and that of course requires dist files.

it is type: library, not drupal-module which I think is related to the issue

The package type in composer.json is unrelated to this.

Would it make sense to try to serve these projects from the packages.drupal.org endpoint, and not rely on packagist?

Probably not. packages.drupal.org is for transitioning Drupal’s info.yml dependencies into composer package dependencies. We don’t want to add additional complexity to make another Composer endpoint that behaves differently when Packagist.org is best at being a Composer endpoint.

Or maybe some other way to avoid being recognized as a Gitlab repo type?

We want Packagist.org to recognize our repositories are GitLab. Knowing that its GitLab allows assuming how to use the archive API to get dist files.

From @saldeak at Packagist: https://github.com/composer/packagist/issues/1464

Ok I've enabled it, and force-updated all packages using this GitLab instance. So you should have dists everywhere now. Please check everything works as expected.. One possible improvement would be getting me a readonly auth token I can set up on our end so as to ensure we don't get rate-limited, but given the small amount of packages maybe it's fine as is too.

Confirmed that made a difference:

  - Downloading drupal/rat (1.0.0)
Downloading https://git.drupalcode.org/api/v4/projects/project%2Frat/repository/archive.zip?sha=28202b02262a39ac8dbbfd43696b67c0c8c46b71

That cleared up all of our issues as well.