[CI] Use latest Elasticsearch release on tests

Created on 11 March 2024, 10 months ago
Updated 12 March 2024, 10 months ago

Problem/Motivation

https://hub.docker.com/_/elasticsearch does not have :latest tag for dockerimages. Maybe we could get the information on simple curl and use "ES_VERSION" as variable in .gitlab-ci.yml file?

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

📌 Task
Status

Active

Version

8.0

Component

Code

Created by

🇫🇮Finland sokru

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @sokru
  • 🇨🇦Canada mparker17 UTC-4

    I think the ElasticSearch maintainers do this because using the "latest" tag means that a Docker image maintainer has to be very careful about the order that they push things to to the registry with: see also IBM: Why can't I pull the newest image by using the latest tag?, James Walker: Understanding Docker's "latest" Tag, and Marc Campbell: The misunderstood Docker tag: latest. Also, running the :latest version of a container is a bit like running against the -dev version of a Drupal module: that is to say, if something breaks, you can't be certain if you made a mistake in your code, or if your code is fine and something changed in ElasticSearch: i.e.: if you use latest, you cannot be sure that you are changing one thing at a time - a common debugging practice.

    That being said, we should consider opting in to run automated tests on future versions of Drupal core (i.e.: use a "testing matrix") so we can catch incompatibilities early; and testing against the latest version of elasticsearch 8 would be good.

    The Docker Hub API says we can get a list of tags for the ElasticSearch image by sending an HTTP GET request to https://hub.docker.com/v2/namespaces/library/repositories/elasticsearch/tags - it returns data in JSON format. If you have jq tool installed, and your version of sort(1) supports version number sort ordering with -V (and ElasticSearch uses SemVer and sort(1) handles semver properly), then...

    curl -sS 'https://hub.docker.com/v2/namespaces/library/repositories/elasticsearch/tags/' | jq -r '.results[].name' | grep -E '^8' | sort -V -r | head -n1
    

    ... should show us the latest version. But, since that requires a shell to run, we run into a sort-of "who came first, the chicken or the egg" problem, where we need a running container in order to tell us which container version to run, and I'm not sure how to do that.

    Perhaps instead, we could add our own custom test job to return an error if the version we specify in .gitlab-ci.yml is the latest... something like...

    variables:
      ES_VERSION: '8.12.2'
    # ...
    .with-elasticsearch: &with-elasticsearch
      - alias: elasticsearch
        # ...
        name: elasticsearch:{$ES_VERSION}
    # ...
    elasticsearch_version_check:
      stage: test
      image: # something that has curl, jq, grep, sort, head, echo, and exit available to use
      script:
        - if [ "$ES_VERSION" != "$(curl -sS 'https://hub.docker.com/v2/namespaces/library/repositories/elasticsearch/tags/' | jq -r '.results[].name' | grep -E '^8' | sort -V -r | head -n1) ]; then
            echo "Testing with an old version of ElasticSearch"
            exit 1
          fi
    
  • 🇫🇮Finland sokru

    One option is to just manually bump the version number every time a new minor is released, but then we rely on our memory/resources to do the change.

    I think many Drupal contrib modules use OPT_IN_TEST_PREVIOUS_MINOR in .gitlab-ci.yml to make sure the module has backward compatibility.

Production build 0.71.5 2024