Include for hidden variables cannot be found in remote GitLab instances

Can you put the code of your integration? How are you including our remote templates?

include:
  - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/include.drupalci.main.yml
    rules:
      - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        when: never
      - if: $CI_COMMIT_TAG
        when: never
      - if: $CI_COMMIT_BRANCH == "develop" || $CI_COMMIT_BRANCH =~ "/^feature\/.*/"
  - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/include.drupalci.variables.yml
    rules:
      - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        when: never
      - if: $CI_COMMIT_TAG
        when: never
      - if: $CI_COMMIT_BRANCH == "develop" || $CI_COMMIT_BRANCH =~ "/^feature\/.*/"
  - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/include.drupalci.workflows.yml
    rules:
      - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        when: never
      - if: $CI_COMMIT_TAG
        when: never
      - if: $CI_COMMIT_BRANCH == "develop" || $CI_COMMIT_BRANCH =~ "/^feature\/.*/"

That helps. I think I know the reason why, but I need to think of the best way without compromising our main use case and approach.

Is it a hard fail or just a warning? If it's just a warning, you could add an extra line for the "hidden" variables file:

  - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/includes/include.drupalci.hidden-variables.yml

It is a hard fail, because the yml file is invalid, so the pipeline does not even start.

Ready for review.

MR: https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/208...
Downstream pipelines (aka projects in our instance): https://git.drupalcode.org/project/gitlab_templates/-/pipelines/175216

You can test this in your remote instance like this.
- Change - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/include.drupalci.variables.yml
- to - remote: https://git.drupalcode.org/issue/gitlab_templates-3447810/-/raw/3447810-include-hidden-variables-via-https-remote/includes/include.drupalci.variables.yml

Just that line should be enough as it's the only file we're modifying in this MR.

If the test doesn't work, then I might need to do some more changes and ask you to try.

Dowstream pipelines after the last change: https://git.drupalcode.org/project/gitlab_templates/-/pipelines/175229

We changed it accordingly. Now we get:

ERROR: Job failed: failed to pull image "drupalci/php--apache:production" with specified policies [always]: Error response from daemon: pull access denied for drupalci/php--apache, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:250:1s)

PHP version is missing.

Ok, that means that the hidden variables file was not included. Let me do another change to the MR.

I made a change to the MR. It's not necessarily what I'd want for the final version, but it's the only way to test. Testing instructions on your end are exactly the same as #7, so hopefully just re-running a pipeline might be enough.

Now we get:

Invalid CI config YAML file
https://git.drupalcode.org//-/raw//includes/includes/include.drupalci.hidden-variables.yml: Invalid configuration format

I made another little change to the way we access the variables, can you try again?

We still get:

Invalid CI config YAML file
https://git.

drupalcode.org//-/raw//includes/includes/include.drupalci.hidden-variables.yml: Invalid configuration format

Ok, I guess that explains why the if/else initial approach didn't work. It's not substituting the variables _GITLAB_TEMPLATES_REPO and _GITLAB_TEMPLATES_REF when inside the file for the include.

I need to think about which other options we have.

Ok, I made it include the file only when we are on our instance, and documented what needs to be done if the templates are used on another instance.

In this case, you need to do an additional include in your file, so taking the syntax in #3 as reference, it'd be something like this:

include:
  - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/include.drupalci.main.yml
    rules:
      - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        when: never
      - if: $CI_COMMIT_TAG
        when: never
      - if: $CI_COMMIT_BRANCH == "develop" || $CI_COMMIT_BRANCH =~ "/^feature\/.*/"
  - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/include.drupalci.variables.yml
    rules:
      - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        when: never
      - if: $CI_COMMIT_TAG
        when: never
      - if: $CI_COMMIT_BRANCH == "develop" || $CI_COMMIT_BRANCH =~ "/^feature\/.*/"
  - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/include.drupalci.hidden-variables.yml
    rules:
      - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        when: never
      - if: $CI_COMMIT_TAG
        when: never
      - if: $CI_COMMIT_BRANCH == "develop" || $CI_COMMIT_BRANCH =~ "/^feature\/.*/"
  - remote: https://git.drupalcode.org/${_GITLAB_TEMPLATES_REPO}/-/raw/${_GITLAB_TEMPLATES_REF}/includes/include.drupalci.workflows.yml
    rules:
      - if: $CI_PIPELINE_SOURCE == "merge_request_event"
        when: never
      - if: $CI_COMMIT_TAG
        when: never
      - if: $CI_COMMIT_BRANCH == "develop" || $CI_COMMIT_BRANCH =~ "/^feature\/.*/"

Note the extra entry for the ...hidden-variables.yml file. Can you try this?

Thank you. This works.

Great! Thanks for confirming.

The above comment covers the scenario where the templates are used in an external instance.
https://git.drupalcode.org/project/keycdn/-/pipelines/177843 shows a pipeline within our own instance.

I've merged this. You can go back to using the main repo for this and not the fork. I will create a release soon after this, but if you have _GITLAB_TEMPLATES_REF: "main" then you should get the change straightaway.

Automatically closed - issue fixed for 2 weeks with no activity.

Automatically closed - issue fixed for 2 weeks with no activity.