Stop changing the meaning of tests

, our tests now fail because "current" now refers to a different version.

I have a lot of issues with how gitlab_templates is being run, however in their defense the default was always suppose to be "the latest release in the latest major" the use of 'current' was a bit of a misnomer that came out of them adding the 'previous major' and 'next major' jobs.

The value was also (originally) suppose to be updated the day of the release (in other words failures should have started months ago unless you chose to change what was originally $_TARGET_CORE and is now $DRUPAL_CORE iirc.

It is possible this was not communicated well although I know blog announcements about the changes being made. It is possible the base documentation needs to make this more clear that the 'current' job is always suppose to be "the latest in the highest released major".

I suggest that if OPT_IN_TEST_SUPPORTED is set to 1, then it tests on all versions of core that are not EOL and that match core_version_requirement.

I've been pondering this lightly in the back of my brain (especially to implement a dynamic min/max test) however I have not sat down to seriously consider the implication details.

The "back of the napkin" design is to do this would require a single job that runs, reads the composer.json, and fires off individually triggered jobs for each combination, or a matrix for every single branch available with a manual trigger (this would have some bad UI)

There are some complexities I have not looked at yet (such as permissions to trigger the job, how we get the data back into the MR if we dynamically generate the jobs etc).

The easier solution for this as noted above to 'lock down' the tested version to what the module supports and not depend upon the existing variables ( use DRUPAL_CORE: ^10 to always get the latest 10.* release, combine with prefer-stable in composer.json to avoid dev releases or alpha/betas)

If it is too hard to do OPT_IN_TEST_SUPPORTED, then I suggest having variables like OPT_IN_TEST_10_3, OPT_IN_TEST_10_4, OPT_IN_TEST_11_0, etc. This way, I can configure it to run tests on versions that are supported. As a project becomes compatible with newer versions, I can add those, but not before.

We've communicated via blog posts ( https://www.drupal.org/drupalorg/blog/gitlab-ci-templates-will-make-drup... → ), issues ( 📌 Update templates so 11.0 is the default/current branch RTBC ) and a good deal of slack channel announcements in #gitlab about the changes.

Drupal itself is a moving target, and as mentioned, the goal of the templates was/is to be able to test against the latest release, without needing to do any changes. Doing the opposite (testing a fixed release) is also possible by changing just one variable.

But in any case, we fully understand that each module or even maintainer has different needs. Because of that, GitLab CI allows for full per-project customizations, we are not limiting almost anything. The templates are just that, templates. They are a starting point to help maintainers with CI tasks that they may or may not know how to set up.

Note that:
- If you want to "pin" the templates to a certain version you can do it as explained here: https://project.pages.drupalcode.org/gitlab_templates/info/templates-ver...
- If you want to create full variants for min or max that adapt to your project, you can do it as explained here: https://project.pages.drupalcode.org/gitlab_templates/info/variants/#cre...
- Last but not least, you can copy the templates files to your projects, and then start changing them at will, having no dependency whatsoever on what's decided on this project.

Creating all those suggested variants in the generic templates would make the project incredibly complex and hard to maintain, especially when you, as a maintainer, can create them easily and fully tailor-made to your needs.

I think that the templates "work as designed" in this case, and that change/evolution of them is inevitable, so I'm closing this issue unless there is a more concrete suggestion that is not implemented or documented already.

I appreciate the feedback in any case.

I don't understand why anyone would want the tests to change on their own so that they start testing on a version of Drupal that they know the module is not compatible with. As it is, I now have to update the testing config in every project I work on. Other developers will have to do the same. My needs are not unusual. The templates should make it easy to do what most people want.

Creating version variants like OPT_IN_TEST_11_0 would be some extra work, but it would only have to be done in one place and every project could use the config. As it is, every developer has to learn how to customize their .gitlab-ci.yml.

I ran into this issue as well with a distro. I pushed up a simple change and the pipeline failed. Previously the "current" version of Drupal was 10.3.x, but now it's 11.x. I was able to fix the issue by adding this to my .gitlab-ci.yml file:

composer:
  variables:
    DRUPAL_CORE: 10.3.6

Hi smulvih2,
The fix you suggest in #6 is OK for now, and it is good that your pipeline is running. But there are alternatives that are better solution. Your change means that you have customised the "current" pipeline thread of jobs and changed it from Drupal 11 to being locked at 10.3.6. This means that other jobs that depends on 'composer' will all be affected, for example all the linting and standards jobs. You may need to customise other variables in future to keep the thread consistent, for example PHP version or DB veresion. Also when Drupal 10.3.7 is release you will have to change your file again if you want to test against that, and then again when 10.4 is released ...

So you may find it better to opt out of running the "current" jobs, but instead chose the "previous major" set. This is currently Drupal 10.3.6 but will be automatically updated to newer Drupal 10 versions when they are released and everything will be consistent for you. So replace your customsation above with:

variables:
  OPT_IN_TEST_CURRENT: 0
  OPT_IN_TEST_PREVIOUS_MAJOR: 1

In this case, his solution is better suited for the context, because it’s a distribution, not a module. So there are no tests to be run, but the linting jobs are still desired.

Linting jobs are only run for the “current” variant, so that fix works well enough for this case.

Doing 10.3.x or 10.4.x would test latest instead of stable but that’s a maintainer’s preference I guess.

Thanks fjgarlin, that's useful background info. I had missed the point that there were no PHPUnit test.