gitlab artifact caching doesn't work due to differences with project ID and build directories

Created on 28 August 2024, 19 days ago
Updated 10 September 2024, 6 days ago

Problem/Motivation

See discussion in https://drupal.slack.com/archives/CGKLP028K/p1724653387328039

CI_PROJECT_ID differs for core test runs depending on whether you're a core committer or not, so we can't use it to determine the target to fetch artifacts from.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

๐Ÿ› Bug report
Status

Needs work

Version

11.0 ๐Ÿ”ฅ

Component
PHPUnitย  โ†’

Last updated about 1 hour ago

Created by

๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @catch
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch
  • Merge request !9352Hard-code the project ID. โ†’ (Open) created by catch
  • Status changed to Needs review 19 days ago
  • Pipeline finished with Success
    19 days ago
    Total: 347s
    #266858
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch
  • Pipeline finished with Failed
    19 days ago
    #267605
  • First commit to issue fork.
  • Merge request !9366Resolve #3470641 "Non committer" โ†’ (Open) created by mstrelan
  • Pipeline finished with Success
    19 days ago
    Total: 534s
    #267720
  • Pipeline finished with Failed
    19 days ago
    #267774
  • Pipeline finished with Success
    19 days ago
    Total: 674s
    #267812
  • Pipeline finished with Success
    19 days ago
    Total: 684s
    #267829
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch

    As well as the project ID being different, we're also running into both the yarn lint and phpstan caches using absolute paths.

    Tried a rewrite as part of the pipeline process and it partially helps (i.e. phpstan fails to use the cache, but gives less criteria it failed by), but not enough:

    https://git.drupalcode.org/issue/drupal-3470641/-/jobs/2585012

    https://git.drupalcode.org/issue/drupal-3470641/-/jobs/2585008

    Possible workaround mentioned here is to use bind mounts:

    https://github.com/phpstan/phpstan/issues/8599#issuecomment-2059227168

    e.g. we'd bind mount the current directory to /tmp/build, cd, do the composer install + run the linting jobs there both for the cache warming job and for the individual linting jobs - that way everything runs in the same directory as far as yarn and phpstan are concerned.

    However to fully test that we'd need to commit the change to HEAD so that the cache warming job runs with the bind mount, then see what the effect on pipelines is after that. Can probably hack the API URLs to point to the pipeline instead of core just to make sure it doesn't break things, but not easy to demonstrate it actually works.

  • Status changed to Needs work 19 days ago
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom catch

    Tried a bind mount but the user in the pipeline doesn't have permission to create one. This might need some help from infrastructure, i.e. can our k8s runner setup allow the bind mount, or can we do something there to have a consistent directory name?

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States cmlara

    Since weโ€™re on docker runners an over simplified possibility

    mkdir /build;
    cp -Ria $CI_BUILD_DIR /build
    cd /build
    <execute as normal>
    cp -Ria anything_you_need_to_archive $CI_PROJECT_DIR 
Production build 0.71.5 2024