Enable distributed caching in GitLab Runner

I've put together an internal MR to enable this. It's based on the setup suggested here https://docs.gitlab.com/runner/configuration/advanced-configuration.html... and using "iam" authentication.

Ran into this too: https://drupal.slack.com/archives/CGKLP028K/p1695219548515259

This can deliver a nice boost in some scenarios! 😊

Bump — this has the potential to help with

11.5 GiB Project Storage

— https://git.drupalcode.org/project/big_pipe_sessionless
🤯11.5 GB!!! 👆 for a trivial module, that is being tested against 9.5, 10.0, 10.1 ,10.2 and 11.0.
This is because of a nightly scheduled build that has to do multiple composer builds, each of which takes ~100 MB: https://git.drupalcode.org/project/big_pipe_sessionless/-/artifacts.

If we extrapolate this to 100s or even 1000s of contrib modules doing multi-version testing (which they should!), then this is going to quickly become untenably expensive AFAICT? (/cc @hestenet (he/him)). Can we automatically clear out all artifacts from all previous scheduled runs, at least the ones that were successful? That’d make a huge difference I think?

— https://drupal.slack.com/archives/CGKLP028K/p1698398728821119

I bet even start using local runner's cache can speed-up pipelines

Simplest approach is just add fallback cache https://docs.gitlab.com/ee/ci/caching/#per-cache-fallback-keys
Then tune-up push/pull policy https://docs.gitlab.com/ee/ci/caching/#use-a-variable-to-control-a-jobs-...

Alternatively, cspell for example (7.5MB) can use artifacts via https://docs.gitlab.com/ee/ci/jobs/job_artifacts.html#with-a-cicd-job-token

Filed new issue as it's the blocker for new infra #3449463: Update DB images to prevent server restart →

Do we have an ETA for this? 🤞

This could AFAICT massively speed up CI runs, and reduce infrastructure costs!

If we don’t implement 🐛 Random HTTP timeouts for GitLab CI jobs Active this issue would be helpful to avoid failing jobs (at an expense of increase in S3 storage usage)

Cache is not a replacement for artifacts, this won’t fix the storage issue mentioned above but can help decrease runtimes by copying some temporary cache files (like phpstan analysis cache) though that will not help
issue forks (cache is per project)

If this doesn't help issue forks, do we want to look at moving phpstan analysis caches into artifacts somehow?

Caching of Composer and NPM dependencies can significantly reduce the pipeline execution time and resources usage, including bandwidth, because now we download packages again from scratch in every pipeline.

Also, it should reduce the load and bandwidth on the Composer repo servers too.

I tried to enable caching in the issue ✨ Implement caching of the Composer packages Active and in a contrib module, but faced this issue, so the GitLab cache is not configured, here is an example: https://git.drupalcode.org/project/commercetools/-/jobs/4701921

Restoring cache
Checking cache for composer-11.1.4-09bfc62449f88e9b34f39b2dd9328bbff5cb5e0a-non_protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
WARNING: Cache file does not exist
Failed to extract cache

But, surprisingly, it can be uploaded well:

Creating cache composer-11.1.4-7bd9c3ecd13a053b2376270a00aa6f55d8c958da-non_protected...
composer.lock: found 1 matching artifact files and directories 
.composer-cache: found 924 matching artifact files and directories 
vendor: found 28005 matching artifact files and directories 
web/core: found 87866 matching artifact files and directories 
web/modules/contrib: found 266 matching artifact files and directories 
web/themes/contrib: found 9057 matching artifact files and directories 
Uploading cache.zip to https://storage.googleapis.com/new-prod-ocean-s3/path/to/prefix/project/108615/composer-11.1.4-7bd9c3ecd13a053b2376270a00aa6f55d8c958da-non_protected 
Created cache

So, maybe we just miss some easy thing to make it work?

As mentioned in #2, there is an MR in the private repo where the gitlab runner is managed to add caching to s3.

The patch is this (a bit trimmed down):

--- a/applications/gitlab-runner/gitlab-runner-app.yaml
+++ b/applications/gitlab-runner/gitlab-runner-app.yaml
@@ -29,6 +29,14 @@ spec:
         runners:
           config: |
             [[runners]]
+              [runners.cache]
+                Type = "s3"
+                Path = "drupalcode"
+                Shared = true
+                [runners.cache.s3]
+                  AuthenticationType = "iam"
+                  BucketName = "runners-cache"
+                  Insecure = false            
               [runners.kubernetes]
                 namespace = "{{.Release.Namespace}}"

It needs infra to set "iam" per https://docs.gitlab.com/runner/configuration/advanced-configuration.html... first.

Considering the propose config change has been waiting since September of 2023:

What is infras opinion on this? Is this something they actually plan to enable or not given its known positives and negatives?

Since the new cluster has been stable, we can move ahead with this.

Caching looks like its opt-in via editing .gitlab-ci.yml, so it's not too much risk destabilizing tests, until we get something using this into the main templates.

I’m not seeing any explicit notes about cached items expiring and being cleaned up, we will want to make sure it's not a runaway cost.

I’m not seeing any explicit notes about cached items expiring and being cleaned up, we will want to make sure it's not a runaway cost.

IIRC Gitlab currently expects that the storage layer will handle this (add lifecycle rules in S3).