Track AI knowledge of Drupal through time

Problem/Motivation

As described here #3349218-18: Training a Large Language Model for Code (Code LLM) - an expert Drupal A.I. → , current state of GPT-4 is quite good at Drupal (but the future state is unclear)

Taken from 🌱 Fully fledged website and module development with Artificial Intelligence Active

There is a need to track how LLM Drupal knowledge change with each iteration of OpenAI and other companies models. Does the quality of answers get better or worse.

Also test which models by which company performs best at Drupal.

Proposed resolution

1. Prepare set of hard, moderately difficult and easy Drupal questions, testing depth and breadth of Drupal knowledge.
2. Evaluate how different models (released throughout the years) perform to answer Drupal questions using https://www.drupal.org/project/ai_evaluations → framework

Remaining tasks

User interface changes

API changes

Data model changes