Add discoverability of requests for AI Test module

Created on 2 July 2025, about 16 hours ago

Problem/Motivation

We have a module currently called AI Test that have some hard coded mocks on some requests and responses to make sure that we can pretend to be an provider. This works well when used in the context of AI core module, since we have the power to change this.

In discussions with the XB AI team we found that they also want to do something similar. But why reinvent the wheel.

The idea is this instead:

  1. In the AI Test we add so we can do discoverability on all installed modules in a directory like "tests/resources/ai_test" where we will look for yaml files in operation_type folders.
  2. These YAML files will have expected request and response given.
  3. If a request comes in on that operation type it should go through all the expected requests and see if it matches up, and if it does it should give back that response.

The files should map to the InputInterfaces and OutputInterfaces of the provider types in Yaml. We could also add that all operation types from now on should have an FromArray and ToArray function to make it simpler.

This means that you can do kernel testing and functional testing on a contrib module, without having to connect to a real LLM or mock responses via Guzzle etc.

Proposed resolution

  • Add FromArray and ToArray on all InputInterfaces and OutputInterfaces we have. Do not add it in the general interface, since we do not want breaking changes.
  • Add a discovery method in the AI Test module that takes operation type as a parameter.
  • Look up all yaml files in the test/resources/ai_test/{operation_type} directory.
  • YAML decode them and put them in a list that is returned.
  • Cache this, if there will come in more then one request.
  • Create a match method, that loops through all the requests in the list and matches with the current request.
  • Return the first match if found
  • If not found, do what we do now.
  • If found, get the response and use FromArray to create a normalized output that the provider returns.

Remaining tasks

Optional: Other details as applicable (e.g., User interface changes, API changes, Data model changes

📌 Task
Status

Active

Version

1.2

Component

AI Test

Created by

🇩🇪Germany marcus_johansson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024