Local Milvus Database with lando or ddev

Created on 10 October 2024, about 1 month ago

Problem/Motivation

My goal is to get Search Api Working with a local milvus installation. In general I'm using a local lando setup for drupal.

Steps to reproduce

I've tried to setup a local Milvus Database and get it to work with milvus provider module.
I tried around with creating the milvus containers for etcd, milvus-standalone, attu and minio inside my lando.yml.
The Milvus and etcd containers are created and started, but exited shortly after, also when manually starting them again.

A second approach was to simply run a docker-compose.yml with for milvus, and then connect it to the lando_bridge_network like here at the bottom:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.2.16
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"
      
  attu:
    container_name: attu
    image: zilliz/attu:v2.2.8
    environment:
      MILVUS_URL: milvus-standalone:19530
    ports:
      - "8000:3000"
    depends_on:
      - "standalone"


networks:
  default:
      name: lando_bridge_network
      external: true
  

docker inspect lando_bridge_network is at least showing all containers (appserver, database, milvus, etcd, etc.)

So in theory, they are inside the same network.

I could access attu and also create collections etc.

So I've been trying to connect Milvus via the drupal backend provider config for milvus, using the docker IP-Address of the container, localhost, "milvus-standalone" etc. together with port 19530.

Always getting "Could not connect to the server."

Running a telnet from inside the app container seems to work:

# telnet milvus-standalone 19530
Trying 172.24.0.17...
Connected to milvus-standalone.
Escape character is '^]'.

Did anyone already succeed with a local installation of milvus using lando or ddev?

💬 Support request
Status

Active

Version

1.0

Component

Providers

Created by

🇩🇪Germany netvidual

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @netvidual
  • 🇩🇪Germany marcus_johansson

    Ah, DDEV I can help you with.

    services:
      etcd:
        container_name: ddev-${DDEV_SITENAME}-etcd
        image: quay.io/coreos/etcd:v3.5.5
        environment:
          - ETCD_AUTO_COMPACTION_MODE=revision
          - ETCD_AUTO_COMPACTION_RETENTION=1000
          - ETCD_QUOTA_BACKEND_BYTES=4294967296
          - ETCD_SNAPSHOT_COUNT=50000
        volumes:
          - ./milvus/volumes/etcd:/etcd
        command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
        healthcheck:
          test: ["CMD", "etcdctl", "endpoint", "health"]
          interval: 30s
          timeout: 20s
          retries: 3
        labels:
          com.ddev.site-name: ${DDEV_SITENAME}
          com.ddev.approot: ${DDEV_APPROOT}
    
      minio:
        container_name: ddev-${DDEV_SITENAME}-minio
        image: minio/minio:RELEASE.2023-03-20T20-16-18Z
        environment:
          MINIO_ACCESS_KEY: minioadmin
          MINIO_SECRET_KEY: minioadmin
        expose:
          - "9001"
          - "9000"
        volumes:
          - ./milvus/volumes/minio:/minio_data
        command: minio server /minio_data --console-address ":9001"
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 20s
          retries: 3
        labels:
          com.ddev.site-name: ${DDEV_SITENAME}
          com.ddev.approot: ${DDEV_APPROOT}
    
      milvus:
        container_name: ddev-${DDEV_SITENAME}-milvus
        image: milvusdb/milvus:v2.4.1
        command: ["milvus", "run", "standalone"]
        security_opt:
        - seccomp:unconfined
        environment:
          ETCD_ENDPOINTS: etcd:2379
          MINIO_ADDRESS: minio:9000
        volumes:
          - ./milvus/volumes/milvus:/var/lib/milvus
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
          interval: 30s
          start_period: 90s
          timeout: 20s
          retries: 3
        expose:
          - "19530"
          - "9091"
        depends_on:
          - "etcd"
          - "minio"
        labels:
          com.ddev.site-name: ${DDEV_SITENAME}
          com.ddev.approot: ${DDEV_APPROOT}
    
      attu:
        container_name: ddev-${DDEV_SITENAME}-attu
        image: zilliz/attu:v2.3.10
        expose:
          - "3000"
        environment:
          - MILVUS_URL=milvus:19530
          - VIRTUAL_HOST=${DDEV_SITENAME}.ddev.site
          - HTTP_EXPOSE=8521:3000
          - HTTPS_EXPOSE=8521:3000
          - SERVER_NAME=${DDEV_SITENAME}.ddev.site
        depends_on:
          - "milvus"
        labels:
          com.ddev.site-name: ${DDEV_SITENAME}
          com.ddev.approot: ${DDEV_APPROOT}
    
  • 🇬🇧United Kingdom scott_euser

    Ddev also has helper commands to do it for you in the VDB provider milvus documentation page (the docker compose needed is in that submodule)

  • 🇩🇪Germany netvidual

    Thanks to both of you, worked fine :-)
    Will try to adapt the config for lando aswell and post it here for refrence if successful.

  • 🇩🇪Germany marcus_johansson

    @netv - would it be ok to close this issue?

  • 🇩🇪Germany netvidual

    Yes, can be closed. Thanks again for your help.

Production build 0.71.5 2024