[Meta] Bring AI modules Tests, Error Handling and Documentation

Created on 3 December 2024, about 2 months ago

Problem/Motivation

For Drupal AI module v1 and the release of Drupal CMS v1 we want to improve the tests, error handling and documentation so it works well for a v1 module. This is a generic issue to specifically keep track of things that need to be done just for v1.

Steps to reproduce

Proposed resolution

Remaining tasks

🌱 Plan
Status

Active

Version

1.0

Component

Miscellaneous

Created by

🇬🇧United Kingdom yautja_cetanu

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @yautja_cetanu
  • 🇬🇧United Kingdom yautja_cetanu

    Things we can do:

    Error Handling

    • Static Analysis
    • Start at a high level and make sure we have it

    Tests

    • Focus on Areas that are most complex or more important (Especially for Drupal CMS)
    • Look at reports of errors we've received and are they commonly happening in similar areas?
    • Agents - All tools should have at least one test
    • Automated tests for validating schemas

    Documentation Personas

    • Sitebuilder - With the existing tools how do I meet what I am trying to achieve. What do we support as a module and how do they configure those things.
    • Developers - How do I use the code and make use of it.

    Site builder questions:

    • How do I set up a provider?
    • Where do I find a list of available providers?
    • How do I set up a "Hello World", Explorer, Automator, AI Search + Boosted Views, AI Search + Chatbot, Assistant + Agents?
    • A page that goes through all configuration options and what they do for every form in the AI module?
    • Examples of set ups
    • Example prompts for different situations
    • Prompt Engineering help with Agents
    • Prompt Engineering help with AI Search + Chatbot

    Developer Questions.

    • What API are there?
    • What does the module do?
    • How do I add new providers, agents, automators, CK Editor Assistants, VDB providers, Chatbots front-ends, Operation Types??, Logging types, various other plugins?
    • What is logged and how it works
    • AI Module Event types that can be used, and how they can be used
    • How do I integrate it with "This part of our site we don't currently support"
    • Hello world, where you module uses the AI module to call an LLM, where you expode a helper form in your module to allow changing models.
    • How do I contribute?
    • How do automated tests work (Unit Tests, Kernel Tests, Tests that need Mock LLM calls, Tests that need real LLM calls)
    • How do I get started locally (Eg DDEV recipes for Milvus, LMStudio)?
    • How do I get evaluations running and test Drupal CMS?

    General Process

    • Where do we put this documentation?
    • Where do we manage the Project Management of the documentation?
    • Where do we manage the Project Management of Tests and Error Handling?
  • 🇬🇧United Kingdom MrDaleSmith

    I would say that:

    • How do I contribute?
    • How do automated tests work (Unit Tests, Kernel Tests, Tests that need Mock LLM calls, Tests that need real LLM calls)
    • How do I get started locally (Eg DDEV recipes for Milvus, LMStudio)?
    • How do I get evaluations running and test Drupal CMS?

    are all questions for Drupal to answer, not one specific contrib module. None of those will have answers that are specific to this module and wouldn't be the kind of thing a maintainer would want to get bogged down keeping up to date.

  • 🇬🇧United Kingdom MrDaleSmith

    "Where do we put this documentation?" is a question that has already been answered.

Production build 0.71.5 2024