Provide development environment for AI development using DDEV

Created on 27 June 2025, about 2 months ago

Description

To develop for Drupal AI we need a unified development environment using DDEV. This will allow developers to develop against the same codebase and configuration.

Similar to Drupal CMS (not using Drupal CMS) we need a setup that any developer can use to contribute and review.

Requirements

Similar to Drupal CMS (see below) We need the following:

  • Composer project to installs Drupal with selected recipes
  • Standard recipes (we could use some from Dupal CMS) that include sensible demo content, so AI functionality can be tested
  • Recipes that install and configure AI modules
  • Documentation how to setup a local development environment

After running the installer a fresh Drupal instance should be setup with a sensible CMS configuration with sensible demo content. AI modules should be already installed, and selected functionality should be pre-configured.

Different AI configurations should live in different recipes. Some recipes should be installed by default. Others should be optional to install.

One or more AI providers should be installed by default. API keys should be easy to configure (maybe during installation) or our demo environment should ship with an readyly configured AI provider (for example Amazee).

Notes (feedback from the maintainers Marcus โ†’ and Artem โ†’ ):

  • Maybe an AI recipy is enough. Instead of having a custom composer project, Drupal's recommended project โ†’ could be used.
  • Only minimal functionality should be installed. Additional recipes that provide AI configurations should be optional to avoid facing regressions in other AI functionality.
  • Realistic demo content and some content types are necessary for AI search, but also important for reviews in general.
  • Recipes from Drupal CMS (see below) can be added, but we should not have dependencies in AI development to avoid facing regressions in un-related Drupal functionality.
  • Amazee only has chat + embeddings. However, we can work with that to start with and have requirements of adding other providers for other task.
  • We should have Postgres vector db, since its easy to export/import from and also is what we will be using with DevPanel.

Examples and links for Drupal CMS

The documentation for setting up Drupal CMS with DDEV can be found here:

The composer project is here:

A recipe including demo content for the first version of Drupal CMS was created
here โ†’ .

Todo

  • Align with Marcus โ†’ because some prior work exist for the integration of Drupal AI with DrupalForge
  • Create a proposal before implementation as issue comment
  • Implement first version
  • List follow up tasks as issue comment

Accept

  • When I follow the instructions in the comment, I can test the DDEV setup and verify setup works.
  • When I setup the site I see a nicely configured Drupal CMS with sensible demo content.
  • When I check the AI providers, I can verify that a provider is pre-configured or I can easily configure them myself.
  • When I check the documentation in the comment, I can test selected AI functionality.
  • Once the instructions and documentation have been verified, they are documented on Drupal.org.
๐Ÿ“Œ Task
Status

Active

Version

1.1

Component

AI Core module

Created by

๐Ÿ‡ฉ๐Ÿ‡ชGermany breidert

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @breidert
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany breidert

    Additional information: At 1xINTERNET we use an bash script to setup a local development environment. For reference I added the docker compose file for Milvus vector database โ†’ and the relevant parts of the installation script.

    # Add RAG support.
    if [ "${rag}" == "true" ]; then
      echo "Enable Milvus vector database"
      mv .ddev/example.docker-compose.milvus.yaml .ddev/docker-compose.milvus.yaml
      ddev restart
    fi
    
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    I uploaded the same for pgvector, it needs the following command to be run inside the web container to work:

    PGPASSWORD=db psql -U db -h localhost -p 5432 -d db -c "CREATE EXTENSION IF NOT EXISTS vector;"

    If its for people contributing, the drupal/core-dev should be installed by default and we should also create ddev commands that people that do not know phpunit well, can run tests. So you can run "ddev test-ai-module" or something similar.

    I would argue that we could also install git hooks for commit, that runs against our grumphp. I'm not sure if that is too "serious"? It would be good though, that stuff like linting gets cought before pushing. Its annoying that I have to merge and fix that a lot of the time.

    We should add so we have one container that sets up mkdocs by default (https://github.com/Metadrop/ddev-mkdocs maybe?), so people can change and watch the documentation.

    All the things I write here does not need to be in version 1.0.0 of the dev environment, but they would all be good things to have.

    Where should the repo be hosted for this or is the idea that its part of the AI module?

  • ๐Ÿ‡ซ๐Ÿ‡ทFrance andypost

    sets up mkdocs by default

    Any reason for that? if you have LLM it knows more then generated docs (which require regular updates and will bring more collisions to MR)

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    Any reason for that? if you have LLM it knows more then generated docs (which require regular updates and will bring more collisions to MR)

    We already have the documentation in the repo, so nothing changes there in terms of MRs.

    It's just to make it easy to contribute code without having to learn how to install and start mkdocs on different operating systems. We can then just say - visit https://mkdocs.ai-playground.ddev.site and any changes to your markdown documentation will autoupdate there on save.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    One thing we could also think about is having Selenium + VNC server on a web url like https://test-viewer.ai-playground.ddev.site, where its possible to watch them play out. Meaning you will see the test start the website, click around and what happens.

    We also have FFMpeg recording possibilities on the Gitlab test suite, so we could add it here as well, though its probably overkill.

    Note: ai-playground is just a placeholder.

  • ๐Ÿ‡จ๐Ÿ‡ฆCanada bisonbleu

    This is just a raw dump of some of the commands I used to create my Drupal AI Ddev sandbox.

    - DDEV
    % mkdir dai-sandbox && cd dai-sandbox
    % ddev config --project-type=drupal11 --docroot=web
    % ddev composer create โ€œdrupal/recommended-project:^11.0.0โ€ --no-install
    % ddev composer require drush/drush:^13.6
    % ddev composer install
    % mkdir -p config/sync
    % cp web/sites/example.settings.local.php web/sites/default/settings.local.php
    - configure settings.php: config_sync_directory, hash_salt, trusted_host_patterns
    % ddev start
    % ddev drush site:install --locale=en --account-name=admin --account-pass=admin -y
    % git init
    - update .gitignore as required e.g. *.sql, *.zip, *.mysql.gz, etc.
    - add useful modules; customize for your requirements
    % ddev composer require drupal/admin_toolbar drupal/pathauto drupal/redirect drupal/token drupal/ctools
    - add ai related modules; customize for your requirements
    % ddev composer require drupal/ai:1.1.x-dev@dev drupal/ai_agents:1.1.x-dev@dev drupal/ai_provider_anthropic:1.1.x-dev@dev drupal/ai_provider_openai:1.1.x-dev@dev
    - to test patches in composer.json
    % ddev composer require cweagans/composer-patches

    - DDEV EXTRAS:
    - Adminer (like PhpMyAdmin but smaller/nicer)
    % ddev get ddev/ddev-adminer
    - Mailpit: captures outgoing emails; itโ€™s bundled in Ddev, you can open it with this command;
    % ddev mailpit

    - PHPUNIT (setup)
    % ddev composer require --dev drupal/core-dev
    % ddev composer update --with-all-dependencies
    % cp web/core/phpunit.xml.dist web/core/phpunit.xml
    - configure as follows

    <env name=โ€œSIMPLETEST_BASE_URLโ€ value=โ€œhttps://dai-sandbox.ddev.siteโ€/>
    	<env name=โ€œSIMPLETEST_DBโ€ value=โ€œmysql://db:db@db/dbโ€/>
    	<parameter name=โ€œoutputDirectoryโ€ value=โ€œsites/simpletest/browser_outputโ€/>

    - then you should be able to run tests, for example:
    % ddev exec โ€œcd web/core && ../../vendor/bin/phpunit tests/Drupal/Tests/Core/Routing/UrlGeneratorTest.phpโ€
    % ddev exec โ€œcd web/core && ../../vendor/bin/phpunit ../modules/contrib/pathauto/tests/src/Unit/VerboseMessengerTest.phpโ€

    - PHP_CodeSniffer / phpcs (setup)
    % ddev composer require --dev drupal/coder
    - register the Drupal coding standards:
    % ddev exec vendor/bin/phpcs --config-set installed_paths vendor/drupal/coder/coder_sniffer
    - check what standards are available (should include โ€œDrupalโ€, โ€œDrupalPracticeโ€)
    % ddev exec vendor/bin/phpcs -i
    - check a specific file or directory:
    % ddev exec vendor/bin/phpcs --standard=Drupal web/modules/custom/your_module
    % ddev exec vendor/bin/phpcs --standard=Drupal,DrupalPractice web/modules/custom/your_module
    - add a phpcs.xml file in your project root

    <?xml version=โ€œ1.0"?>
    	<ruleset name=โ€œCustom Drupal Standardsโ€>
    	  <description>Custom coding standards for this project</description>
    	  <rule ref=โ€œDrupalโ€/>
    	  <rule ref=โ€œDrupalPracticeโ€/>
    	 
    	  <!-- Paths to check -->
    	  <file>web/modules/custom</file>
    	 
    	  <!-- Exclude certain files -->
    	  <exclude-pattern>*.min.js</exclude-pattern>
    	  <exclude-pattern>*.min.css</exclude-pattern>
    	</ruleset>

    - then you can run this command
    % ddev exec vendor/bin/phpcs

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States darren oh Lakeland, Florida

    This will need its own place to live, since itโ€™s not a module. Should it be a separate project? Iโ€™d volunteer to create a repo for it in the Drupal Forge organization on GitHub, but first I need to hear what everyone else thinks.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany breidert

    @darren oh

    On Drupal.org you can add general projects. Here is the documentation for general projects โ†’ .

    These can contain arbitrary code as for example DDEV installation scripts.

    They are also used for recipes (for example this one โ†’ which we contributed earlier this year).

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States matthews Colorado

    Amazee only has chat + embeddings. However, we can work with that to start with and have requirements of adding other providers for other task.

    What other capabilities would you like to see for the amazee AI provider module? Are you looking for image or video generation? We can add additional models.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany breidert

    @matthews - in the end we need providers listed for the different model types on /admin/config/ai/settings (Image Classification, Image to Video, Speech to Text, etc.).

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    @metthews & @breidert - we have some of them being very niche and I do not think those are needed. Speech-to-Speech being only Elevenlabs, Audio-to-Audio being only Auphonic. In both those cases the underlying services do not allow for shared accounts, so there is no way Amazee can provide this without breaking the terms and conditions. Also these operation types are niche and the usages for them are for very specific products.

    I think the following are general operation types that are generally useful:

    • Chat (with tools and vision) - already available in Amazee
    • Embeddings - already available in Amazee
    • Moderation
    • Text-To-Image
    • Text-To-Speech
    • Speech-To-Text

    With all these, the more languages they handle, the better. OpenAI is the golden standard handling around 58 languages - but having the major Drupal languages markets would be good I assume?

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    I think another thing we need to add is a document loader - I have attached an Unstructured docker composer file that works with ddev, but we could also use Roberto's extension here: https://github.com/robertoperuzzo/ddev-unstructured

    This might need to be discussed, because โœจ Create Document Loader Normalization Layer Active is still not done, these are at the moment not interchangeable, like the vector database or AI provider is. Meaning you create specific recipes/solutions that rely on a specific service.

    I would however argue that Unstructured is the best, because:

    1. It is open source and can be self hosted.
    2. It handles almost any file format - see: https://docs.unstructured.io/ui/supported-file-types
    3. Its model is best in class at understanding how PDF's are structured and showing it in the right order in markdown.
    4. The Drupal solution is pretty extensive and have more maintainers then me: https://www.drupal.org/project/unstructured โ†’
    5. Unstructured offers SaaS as well, which only requires you to add a api key to use. So people using simple hosters can still use them - just not in a privacy compliant manner.
    6. Any solution will require server applications/changes, unless we solve each of these file formats in PHP - something that will require maintainers we don't have right now.
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    Here are other future proofing applications that should be added to webimage_extra_packages:

    • ffmpeg
    • chromium
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    oh, I think its already there - but python3 and pip3 we should have.

  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    I think that https://github.com/ddev/ddev-drupal-contrib can be a good starting point for this issue.
    Then we can add all required ddev addons and initial packages.
    Finally, we need a way to install Drupal and install the required recipes

    I can work on it :-)

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    I do run nginx with "fastcgi_buffering off;" to make streaming work in all cases. I'm not sure how important that is for a development environment though?

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    @lussoluca - just so its confirmed here as well, that's a great starting point!

  • Pipeline finished with Canceled
    about 1 month ago
    Total: 89s
    #538029
  • Pipeline finished with Canceled
    about 1 month ago
    Total: 217s
    #538028
  • Pipeline finished with Success
    about 1 month ago
    Total: 209s
    #538030
  • Pipeline finished with Success
    about 1 month ago
    Total: 195s
    #538051
  • Pipeline finished with Canceled
    about 1 month ago
    Total: 127s
    #538073
  • Pipeline finished with Failed
    about 1 month ago
    Total: 244s
    #538076
  • Pipeline finished with Failed
    about 1 month ago
    Total: 168s
    #538140
  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    I've prepared a quick recording of a first draft implementation to check if I'm in the right direction.

    I'm using the drupal_cms_ai recipe, but maybe it's too much; should we create an ad hoc recipe for this task?

    (I still have to check why I have a PHPUnit failure...)

  • Pipeline finished with Failed
    about 1 month ago
    Total: 283s
    #538825
  • Pipeline finished with Failed
    about 1 month ago
    Total: 169s
    #538837
  • Pipeline finished with Failed
    about 1 month ago
    Total: 141s
    #538841
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    One thing in regards for mkdocs - we are adding some underlying python project for it to look better. You can see it here for gitlab: https://git.drupalcode.org/project/ai/-/blob/1.2.x/.gitlab-ci.yml?ref_ty...

    We should put this in a script/file so we only have to change this in one place, so any changes to gitlab, changes the local dev environment and the opposite.

  • Pipeline finished with Success
    about 1 month ago
    Total: 203s
    #538873
  • Pipeline finished with Failed
    about 1 month ago
    Total: 143s
    #538877
  • Pipeline finished with Failed
    about 1 month ago
    Total: 137s
    #538878
  • Pipeline finished with Failed
    about 1 month ago
    Total: 138s
    #538912
  • Pipeline finished with Failed
    about 1 month ago
    Total: 134s
    #539129
  • Pipeline finished with Failed
    about 1 month ago
    Total: 165s
    #540782
  • Pipeline finished with Failed
    about 1 month ago
    Total: 257s
    #540785
  • Pipeline finished with Failed
    about 1 month ago
    Total: 262s
    #540894
  • Pipeline finished with Success
    about 1 month ago
    Total: 197s
    #540905
  • Pipeline finished with Success
    about 1 month ago
    #540928
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States rfay Palisade, CO, USA

    I rarely recommend adding a .ddev/config.yaml, and never recommend adding the `name` element, since it means you've wired the name of the project. (It can be omitted)

    The best approaches are DDEV add-ons like https://github.com/ddev/ddev-drupal-contrib

    Just committing everything like this is really inflexible, compared to an add-on where it can be maintained properly and updated easily.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States rfay Palisade, CO, USA

    Related conversation in search_api: https://www.drupal.org/project/search_api/issues/3520659#comment-16149749 โœจ Enabled local development env with DDEV Active

  • ๐Ÿ‡จ๐Ÿ‡ญSwitzerland dan2k3k4 Zurich

    Added a comment to the MR to remove the `name` key

  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    Thanks!

    I think that we have to decide which pattern we want to implement:

    At first, I thought that the first pattern was the one the community was going towards (for example, the Devel module uses it).
    However, it is more complex to maintain and adds several unrelated files to the project. So now I'm more in favor of the second.

    We can develop a new addon, like ddev-drupal-ai-dev, and add a CONTRIBUTING.md file to this repo. explaining how to use it.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    Started adding comments, but saw that you were just asking for information about if its on the right way.

    I think its on the right way - some things are missing, but that was mentioned.

    I realized for pgvector and unstrcutred we need to set this up as well via the recipe.

    I think the Drupal CMS AI might be overkill, we need something that sets up providers and the underlying structure - and I think the other functionalities should be fetched from external playground recipes if you want examples on what you can setup with the AI module.

  • Pipeline finished with Success
    about 1 month ago
    Total: 206s
    #542797
  • Pipeline finished with Success
    about 1 month ago
    Total: 196s
    #542830
  • Pipeline finished with Canceled
    about 1 month ago
    Total: 92s
    #544770
  • Pipeline finished with Success
    about 1 month ago
    Total: 208s
    #544771
  • Pipeline finished with Success
    about 1 month ago
    Total: 182s
    #544841
  • Pipeline finished with Success
    about 1 month ago
    Total: 200s
    #544943
  • Pipeline finished with Failed
    about 1 month ago
    Total: 308s
    #545281
  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    I came up with a solution that is more maintainable and powerful (and more in line with what Randy says).

    I created a new DDEV add-on for that, but it's designed to be used by every contrib module, not only by the ai module: https://github.com/lussoluca/ddev-drupal-contrib-root (of course, the namespace is temporary; we can place it where we want).

    With this new add-on, the only change in the ai module is a CONTRIBUTING.md file: https://git.drupalcode.org/project/ai/-/merge_requests/745/diffs

    Pros:

    • No need to commit anything in this repo, we can update the stack without the need for a new AI release
    • Support for working on multiple modules. Useful when a dev needs to work on both ai and ai_agents at the same time (for example)
    • Support for adding temporary modules or recipes without impacting the composer.json file of the ai module
    • We can then write a specific DDEV add-on to support specific use-cases, like the need for a vector database

    I've to finish it, but I think this can be the solution.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States lpeabody

    Throwing my hat in the ring for supporting a Drupal-centric AI DDEV addon that brings in Milvus by default and can optionally be configured to bring in additional Docker services that can provide, for example, Llama, and can be used by the new AI Docker Provider. DDEV supports direct download of distribution packages:

    e.g. ddev add-on get https://github.com/ddev/ddev-drupal-solr/archive/refs/tags/v1.2.3.tar.gz

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    @lpeabody - the biggest issue I found with Milvus is that its very hard to export and import data. Postgresql is just a sql.gz dump that you can move to the next machine. So if you embed 10000 items on one dev environment you just dump the normal database and the vector database and import it on the other, to not have to do hours of embeddings calls.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    I have one more ask when this is done Luca, but its a big one and not crucial, so I will do a follow up issue. Mockoon so we can mock OpenAI or some other providers, when developing.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States lpeabody

    I've spun up https://github.com/lpeabody/ddev-ai. It re-uses a Docker Compose template that Marcus provided, I found a while back in https://www.drupal.org/project/ai/issues/3479943#comment-15809663 ๐Ÿ’ฌ Local Milvus Database with lando or ddev Active

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States lpeabody

    @lpeabody - the biggest issue I found with Milvus is that its very hard to export and import data. Postgresql is just a sql.gz dump that you can move to the next machine. So if you embed 10000 items on one dev environment you just dump the normal database and the vector database and import it on the other, to not have to do hours of embeddings calls.

    I'll research options for managing Milvus data. I don't know much about it, but if it doesn't have a native backup/import mechanism then an alternative could be using Docker volumes where you want to use Milvus in an other project, but initialize with Milvus data from another by copying the volume used in the other project. Something to experiment with. I'll add a tracking issue.

    Also managed Postgres hosting is easy to come by - with Milvus you kind of need Zilliz or use cloud vendors.

    I'm really approaching the add-on from a local-only perspective. There's no implication that because you're using Milvus locally that you also have to use Zilliz in production. I usually use Milvus locally and Pinecone or OpenSearch in production, with the Search API backend configuration difference being handled by Configuration Split or hardcoded configuration overrides in a settings.php file.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    @lpeabody - currently we have a problem with that, but that should be fixed by 1.2.0. We can't build recipes that are VDB provider agnostic. But that should be fixed around the same time that this comes out. Issue here: https://www.drupal.org/project/ai/issues/3531028 โœจ Make vector databases abstracted and installable for recipes - Part 2 Active

    But there is another reason - one of the reasons to have this development environment is also to be able to quickly spin up recipes with added database dumps for Drupal Forge and Amazee.io for one-click "test out the AI module with free credits" and we have mainly used Postgres there because it was easy to create provisioning scripts for this and it also as mentioned makes the import/export.

    But maybe we could make it a possible to switch using a config value, similar to how you swap it out easily with the normal database?

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    Sorry, just saw your comment - nvm the comments I added.

    +1 for having the development environment outside of the AI module. And the idea is very sound.

    I don't think we should use recipes there, but a ddev command that can clone, install and setup the most popular providers and helper modules like Unstructured.

    Recipes should then be used for adding AI functionality.

  • Pipeline finished with Failed
    29 days ago
    Total: 204s
    #546860
  • Pipeline finished with Success
    29 days ago
    Total: 187s
    #546861
  • Pipeline finished with Success
    29 days ago
    Total: 185s
    #546933
  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    @lpeabody cool,
    We can add the instruction to apply the DDEV AI add-on here: https://git.drupalcode.org/project/ai/-/merge_requests/745/diffs#3f454a9...

  • Pipeline finished with Success
    28 days ago
    Total: 329s
    #547580
  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom andrewbelcher

    Can I recommend https://github.com/ddev/ddev-drupal-contrib as a worthwhile consideration. It does a lot of the effort it setting up an environment that can be flexibly used across different core versions and maintain relative parity with Drupal's CI testing.

    It's slightly geared towards having a single primary module, though I think there are ways that could be handled by making use of require-dev and installer paths to put all the "active" projects into web/modules/custom so the commands like ddev phpunit behave as expected.

    I think that would allow us to do all the config in the main AI module, rather than needing additional projects etc.

  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    https://github.com/ddev/ddev-drupal-contrib is a very good project and I've used it in all my contrib modules (monolog, webprofiler, ...).

    We need to think about pros and cons (in my opinion):

    Pros:

    • Well maintained
    • Parity with Drupal's CI testing

    Cons:

    • ddev-drupal-contrib suppose you clone only one project at a time. At the moment, cloning multiple modules is not supported (you have to do it manually, by going into web/modules/contrib and run a git clone)
    • if you need some other modules (not the source, just to be installed), you can't, unless they're listed in the require or require-dev section of the module's composer.json
    • ddev-drupal-contrib works better if you commit all the DDEV artifacts it creates. The AI ecosystem is huge, here we're thinking about the AI module, but then we should configure every other module in the same way
    • If we commit DDEV artifacts created by ddev-drupal-contrib, then we have to maintain them (update them every time they change on the source add-on)

    Of course, we can find a way to resolve all those issues, but I think that, in the end, this solution will be more complex to maintain and provide a worse DX.

    For example, one request from Marcus was:

    Its very common that you do not just work on the AI module, but also some contrib module. It would make it easier if I for instance want to work on the OpenAI provider that it gets it from repo sources, so you don't have to uninstall, git clone and reinstall it

    The solution proposed in this MR resolves all the issues and will allow us to release new versions of the stack without the need to release a new version of the AI module.

    Just to recap, my proposal is:

    In the end, for me, it's the same. I'll let Marcus and the community decide :-)

  • ๐Ÿ‡ฌ๐Ÿ‡งUnited Kingdom andrewbelcher

    Firstly to say that I don't mean to be putting down the work you've done! I agree that the community, and in particular the maintainers who will end up responsible for it, are the ones to decide. I think both options have good merit and I'm introducing this only to help us find the best overall solution. It may be that we're best to bring in some of the good bits from ddev-drupal-contrib into our own 'custom rolled' stuff.

    I've opened https://github.com/ddev/ddev-drupal-contrib/issues/135 to discuss where I feel the limitations of ddev-drupal-contrib lie. If I've missed anything, do say!

    Your first two cons are the first two points of consideration on there.

    ddev-drupal-contrib works better if you commit all the DDEV artifacts it creates. The AI ecosystem is huge, here we're thinking about the AI module, but then we should configure every other module in the same way

    I'm leaning the other way - we only commit the .ddev configuration for the AI module, and everything else gets worked from in that project.

    If we commit DDEV artifacts created by ddev-drupal-contrib, then we have to maintain them (update them every time they change on the source add-on)

    We only need to update if there is something we want to update for. Once added to a project, the add-ons can be used "as is" without updates/changes.

    An update is just ddev add-on get ddev/ddev-drupal-contrib and then review/commit any changes. Where as the alternative is the AI maintainers taking on not just the maintenance of 3 additional projects (plus documentation), but also the development to provide the functionality we'd want.

    I think the biggest issue to solve for ddev-drupal-contrib is the one of additional modules/dependencies. I feel these fit into 4 categories:
    1. Production dependencies - these go into require, and we can use installer-paths if we want them in web/modules/custom so we can develop them.
    2. Non-production dependencies - these go into require-dev and likewise can use installer-paths if we want.
    3. Things we will always want when developing AI module - these could go in require-dev, but that would mean they also get installed in Drupal CI. This may be an acceptable trade-off, or we may want to find an alternative approach (git submodules could even be an acceptable approach).
    4. An easy way to add ad-hoc dependencies in a specific instance - this could be solved with require-dev and no committing the changes, but that doesn't feel ideal. You could just check them out (and it'd be easy to provide some commands to make that easier).

    I think looking at how preferred-install could be used to make it easier to get working repos would be worthwhile.

    I do think a lot of those could be solved by having a single AI development project, which I think wouldn't need to be more than .ddev and composer.json (and probably a README.md).

  • Pipeline finished with Failed
    27 days ago
    Total: 116s
    #549303
  • Pipeline finished with Failed
    27 days ago
    Total: 117s
    #549311
  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    @andrewbelcher, this is an interesting approach!

    I'm pushing a new MR (https://git.drupalcode.org/project/ai/-/merge_requests/762) with ddev-drupal-contrib artifacts committed and a new command that uses composer require to add a new module to the codebase (similar to ddev poser), for now it is called ai-get-module.

    After setting up an environment:

    1. clone ai module
    2. ddev start
    3. ddev poser
    4. ddev symlink-project

    If I want to add ai_agents (as a git cloned repo) to work on it, I can run:

    ddev ai-get-module drupal/ai_agents:1.1.x-dev@dev from-source

    If I just need the module:

    ddev ai-get-module drupal/ai_agents:1.1.1

  • Pipeline finished with Failed
    27 days ago
    Total: 143s
    #549325
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States moshe weitzman Boston, MA

    Hi all. Lots of good discussion here. I'm happy to discuss and hopefully find a way for this ecosystem to build on top of ddev-drupal-contrib (I'm the maintainer).

    The latest MR from @lussoluca looks great to me.

    I think .ai-get-module could go into ddev-drupal-contrib in some fashion. Maybe it becomes `ddev poser require`. If no second argument is provided, we assume install for backwards compat with current poser. Until then, its fine for ai-get-module to live in this module.

  • Pipeline finished with Failed
    22 days ago
    Total: 186s
    #552798
  • Pipeline finished with Failed
    22 days ago
    Total: 297s
    #552907
  • Pipeline finished with Success
    22 days ago
    Total: 327s
    #552918
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany breidert

    I tested commands the MR, and it works after fixing one small error (I left a comment in the MR).

    mkdir ai-dev && cd ai-dev
    ddev config --project-type=drupal11 --docroot=web --php-version=8.3 --corepack-enable
    ddev add-on get lussoluca/ddev-drupal-suite
    ddev drupal-init 11.2.2
    ddev drupal-get-module ai 1.2.x
    ddev drupal-get-recipe ai_dev_recipe 1.0.x
    

    The recipe applies nicely. I like how you are asked during installation to enter your keys for Anthropic and OpenAI.

  • Pipeline finished with Success
    22 days ago
    Total: 326s
    #552924
  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    Thanks, Christoph.
    That MR is for the version without DDEV committed. Then there is this one for the version with DDEV committed.

    Feel free to review it as well, if you have time.

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany marcus_johansson

    First of testing:
    So running the external one works very well as Christoph mentioned.

    For the internal one, the ai-get-module command works well, its a little bit confusing to have a git repo in a git repo, because its in .gitignore folders, so its darkened out in VSCode at least. It gets installed in the contrib folder for me, but that is just a good thing if a recipe already installed it and enabled it, so it doesn't change directory.

    It is weird to have the AI as the root, when you might be a contributor of a contrib module, of which you are only interested in developing. But that might not be a huge issue.

    I will set this to RTBC, because I think technically its ok and a good foundation for both solutions and follow up issues can be used for details.

    In general
    In terms of maintainership, the idea of reusing ddev/ddev-drupal-contrib is very appealing and specifically if they add the possibility to handle multiple custom modules.

    I would say that the lussoluca/ddev-drupal-suite (in another namespace, with multiple contributors) + some AI specific add-ons is more intuitive, compared to having it inside the AI module. I have looked through some of the major modules and have yet seen one that is actually shipped with a .ddev folder. Managing this in the AI module is a harder chore, than in its own repo. So my decision, if what I write above is true, is to use an external setup.

    I do think in the future that this also makes the switch to ddev/ddev-drupal-contrib easier if it adds features, since lussoluca/ddev-drupal-suite, is already built on top of it. A lot of that can be added with add-ons on top of the ddev-drupal-contrib.

    That still leads to two questions:

    1.
    A initial recipe or command that can setup the providers. The problem with recipes and this is that we have 20+ providers at the moment, and you can't have a terminal setup where you have to give or skip 20-40 different steps to setup the providers you want. You also don't want those 20 projects in your modules directory.

    So there are some solutions I can see to this:
    a. We set it up with amazee, since they offer free credits for developers. Any operation type they do not offer, you have to setup manually or via recipe afterwards.
    b. We create a drush command for settings up the most common providers, where you can select the provider you want - if we assume that this is for developers, a terminal command would be ok. If its drush its allowed to run composer. But this has to be maintained, with updates when it happens.
    c. The same, but a ddev command in the above, but makes it harder to move to ddev/ddev-drupal-contrib.
    d. Document how you get started, but this might be annoying for developers that doesn't want to help with the AI modules, but wants to create some third party feature on top of it.

    2.
    Where do we host this - Drupal General Projects, some github owned by the AI Initiatve etc?

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States rfay Palisade, CO, USA

    I'm fine with whatever you come up with to see how it works out. This is not critical stuff.

    But please, please, leave the `name` element out of the config.yaml so you don't have the wired project name in there. It's far better IMO for it to take the name from the enclosing directory.

  • ๐Ÿ‡ฎ๐Ÿ‡นItaly lussoluca Italy

    But please, please, leave the `name` element out of the config.yaml so you don't have the weird project name in there. It's far better IMO for it to take the name from the enclosing directory.

    Latest config.yaml is correct, right?

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States rfay Palisade, CO, USA

    Yes, that link is fine. It's completely baffling to have two competing MRs and have an issue be marked RTBC without reference (that I understand) to which MR :)

  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany breidert

    @lussoluca after speeking with @marcus_johansson we prefer the addon version. Please close the .ddev MR.

  • Pipeline finished with Success
    22 days ago
    #553136
  • First commit to issue fork.
  • Pipeline finished with Failed
    13 days ago
    Total: 197s
    #560407
  • ๐Ÿ‡ฎ๐Ÿ‡นItaly robertoperuzzo ๐Ÿ‡ฎ๐Ÿ‡น Tezze sul Brenta, VI

    When contributing to fix an issue, we usually need to clone the corresponding forked issue branch on GitLab
    (for example: this one). Because of this, I think the ddev drupal-get-module command might benefit from an additional parameterโ€”not just the "version," but also one to specify the issue forked branch. That way, contributors (not only maintainers) could more easily work on open issues.

    Maybe Iโ€™m stretching the purpose of the ddev command a bit, but I find it really helpful to have two remotes set up like this:

    ai-3532795  git@git.drupal.org:issue/ai-3532795.git (fetch)
    ai-3532795  git@git.drupal.org:issue/ai-3532795.git (push)
    origin      git@git.drupal.org:project/ai.git (fetch)
    origin      git@git.drupal.org:project/ai.git (push)
    
    • origin: so I can rebase my branch on 1.2.x and pull in the latest community changes.
    • ai-3532795: so I can push my own changes to the issue fork.

    What do you think?

  • Pipeline finished with Success
    12 days ago
    Total: 337s
    #561368
  • ๐Ÿ‡ฎ๐Ÿ‡นItaly robertoperuzzo ๐Ÿ‡ฎ๐Ÿ‡น Tezze sul Brenta, VI

    After the today's meeting, I share the two main points of DDEV add-on discussion.

    1. Move ddev-drupal-suite to the DDEV Namespace

    Objective

    Transfer ownership of the
    ddev-drupal-suite add-on
    from the current repository to the official DDEV namespace for greater visibility, credibility, and community adoption.

    Actions

    1. Initial Contact with Randy
      • Prepare a short Slack message or GitHub issue to Randy (maintainer of DDEV).
      • Clearly state the goal: to move the repository under the official ddev namespace.
      • Highlight the benefits: centralized maintenance, broader adoption, consistency with other official add-ons.
    2. Evaluation of Add-on Name
      • Assess whether to keep the current name ddev-drupal-suite or rename it (e.g., ddev-drupal-tools or another clearer alternative).
      • Consider factors such as:
        • Scope of functionality
        • Consistency with other DDEV add-ons
        • Clarity for new users
    3. Repository Transfer Process
      • Ensure the current repo is up-to-date, with a clean commit history and a clear README.md.
      • Add or review a CONTRIBUTING.md to make it community-friendly.
      • Request Randy (or DDEV maintainers) to initiate the transfer.
      • Update all documentation links, CI/CD configurations, and references to point to the new namespace.
    4. Community Announcement
      • Once transferred, announce the move in DDEV channels (Slack, GitHub Discussions, Mastodon, etc.).
      • Include a migration note if repo URLs change.

    2. DDEV Drupal AI Add-on

    Objective

    Create a ddev-drupal-ai add-on that orchestrates AI-related tools and workflows,
    starting from the current proposal in the ddev-drupal-suite repository.

    Scope

    • Provide a glue add-on that centralizes commands for AI workflows.
    • Allow users to:
      1. Request additional add-ons (e.g., unstructured, pgvector).
      2. Install AI "recipes" through an interactive workflow (provider โ†’ functionality โ†’ setup).
      3. Include a new official add-on for pgvector (currently missing).

    Architecture Overview

    • Glue Add-on (ddev-drupal-ai)
      • Acts as a meta-tool.
      • Provides DDEV CLI commands for interactive setup.
      • Manages dependencies and recipes.
    • Add-on Dependencies
      • unstructured add-on (existing).
      • pgvector add-on (to be created).
    • Workflow
      ddev drupal-ai setup
       โ””โ”€ Step 1: Choose AI provider (e.g., OpenAI, Local LLM)
       โ””โ”€ Step 2: Choose functionality (search, embeddings, Q&A, etc.)
       โ””โ”€ Step 3: Install required add-ons (pgvector, unstructured, etc.)
       โ””โ”€ Step 4: Configure environment variables and Docker services
          
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States rfay Palisade, CO, USA

    Transfer ownership of the ddev-drupal-suite add-on from the current repository to the official DDEV namespace for greater visibility, credibility, and community adoption.

    We certainly appreciate add-ons maturing into the ddev namespace, but not until they have a track record of usage and maintenance.

    A casual look says that ddev-drupal-suite doesn't even have the `ddev-get` topic yet, doesn't have tests, and I don't think the README even explains why it exists and doesn't have a comparison to other (mature and existing) add-ons for the drupal world. It looks like this add-on wasn't created from the template, and doesn't subscribe to the best practices expounded in https://ddev.com/blog/ddev-add-on-maintenance-guide

    (I've subscribed to all activity in the add-on)

Production build 0.71.5 2024