Extend Logging Capabilities

Ok firstly this is amazingly cool!!!
So the good things we need to do are:

• Maybe create a video explaining it.
• Maybe a blog post showing screenshots of each page and why its useful and how to use it.
• Would love to bring emma into it and see what she thinks
• Tried downloading a prompt and used it in playground and it gave roughly the same answer as in drupal. Super cool for figuring out what is going on.
• Connect this with Evaluations.
• Have a button on DeepChat (There needs to be a setting to turn this on) that allows you to highlight an assistant message and go to the log directly.

Couple of points:

• What happens if people don't want to spend money on API calls to summarise any log. Is there a setting to turn this off and is there a fall back?
• It would be good to be able to export the log itself and see a similarly well structured log file that is human readable but also machine readable. I'd like it so that multiple front-ends could visualise the logs differently. (For example a react tree of agents, minikanban could replay the logs)
• I'm personally finding myself lost with the "Child" logs and where the one I'm looking at fits in and relates to the child logs. Don't have an immediate solution for it yet

Definitely todo:

Fallback for no LLM summaries.

- Requested Actions - Goes to> "First few lines of last user prompt"
- Performaed action- Goes to> "First few lines of that is the body of the response"

Follow up issues:

Explore how good smaller cheaper models are
Explore embedding a small opensource llm to handle this.

- Merge AI Logging LLM settings into the AI logging settings
- Allow a setting that turns off AI summarisation of logs
- If the setting is turned off, put a warning on the report page "This log overview page is significantly improved with AI summaries that can be turned on here"

Pushed these changes to a new branch 3493277-extend-logging-capabilities-v2

First off, it looks really nice! I added some comments outside of the stuff phpcs etc. found.

The biggest issue right now seems to be that all other operation types logging are gone? While some of them do not make sense at the moment, moderation and embeddings for isntance probably could do, to find out why something got banned or if something was searched for.

Also it adds Performed Actions and Last Performed Actions on any chat log, which also makes little sense, even if it doesn't show it?

Most of my comments are NIT - but I think we need to have a bundle for other operation types in there, otherwise we don't have feature parity?

scott_euser → changed the visibility of the branch 3493277-extend-logging-capabilities-v2 to hidden.

Unfortunately I got blocked before I could actually see the converted logs from existing site. I worked around some of the errors to keep going but got stuck when the update hooks seemed to remove the field storage definitions from existing logs and fail to create the new ones needed by the view.

For as far as I did manage to get I added comments to the MR.

Thanks!

Thanks, I am in the middle of rewrite of the issue code so it plays nicely with the existing logging.

seogow → closed merge request !504

seogow → closed merge request !471

@andrewbelcher

• The change in the AI Providers (in their abstraction) is unfortunately necessary. There was missing an event triggered on streamed response reception.
• 'Prompt' can be a misnomer here - it is an initial request to LLM, commonly named 'Prompt'. That is why you see the dialogue here sometimes and sometimes not - the LLM receives all the dialugue content + System prompt on input. It is not a summary, it is really just humanly styled LLM input (that is why you can download it and copy/paste into LLM and it still works).
• Response should probably be renamed to Chain Response and a new tab Call response should be introduced, to distinguish between response of the whole Call Chain (which is indeed response of its last call - the final reaction to a human query) and response of the actual LLM call (which frequently is an invocation of a function call 'Function call for answer').
• With any complexity of nested calls, the chain will always be uninterrupted, beginning with a human request and ending with LLM providing a final response. The only UX/UI change I would introduce is to make a function decription more prominent, so you wuld see in the log exactly what was called, along with its arguments.
• We should do async processing of the logs, yes, both in Cron and in the batch.
• You do have RAW input - see the 'Prompt' above. the Prompt you download is the raw input. But we can/should also save JSON paypload to LLM.
• Requested and Performed action tabs are intended to show LLM-processed input and output. I do on runtime replace that information with raw data if the processed ones aren't available (for logs recorded before LLM processing was enabled), but I see this is confusing. I will hide the tabs if LLM processed data aren't there.
• Vertical tabs are not natively utilized in the default display of entity views, they are designed to enhance the user experience during content creation and editing. They work on forms, not on display. It would be probably overkill to render a form just to get tabs.
• Having real menu and real fields is an architectural decision stemming from us originally having ability to edit Log entities (since the very first version of the module). I would also prefer to do it programmaticly as the practical advantage for logging is not clear.

Sorry, I forgot to write this here regarding your first bullet point, we discussed it last week - regarding the breaking change.Even if the previous solution that is in the AI module at the moment is not as elegant, it does do streamed logging without needing breaking changes. Since it is a 1.1.x update, we should keep it with as few breaking changes as possible. This means that older providers can work with the newer AI module as well.

The actual solution you can see here, for instance what you would need for OpenAI provider to update to have it working: https://git.drupalcode.org/project/ai_provider_openai/-/blob/3489846-add...

This requires you to always use this method when you yield, but it does not add breaking changes.

The update now seems fine, my View doesn't get broken any more, thanks! Some comments:

1. I assume I shouldn't get this double raw + processed logs
2. We need some sort of help text here explaining what these things are... what does 'processed mean'? I go to it, its empty. I guessed maybe I need to mark something as processed (like reviewed)? but I open a log and I don't see such a button, nor in dropbutton, so I hit a dead end there
3. Is tabs the right interface? The typical interface for this would be Local Tasks I believe, like attached. In which case 'Delete' could also be a local task like it is for Drupal Core database logs rather than the more scary danger button?

Screenshot of what I see after upgrade:

Screenshot of local tasks in database log from core:

@marcus_johansson yes, we discussed that.

The issue with adding event trigger for streaming response into every LLM provider is, that functionality of this event then depends on 3rd party implementations.

Not sure we want to rely upon these, especially with ever growing number of them.

Hence, I believe what we have now is a bug and it should be rectified asap, even if it indeed is a breaking change? Especially when we have just a few LLM providers which do stream at the present time?

Thanks @scott_euser!

1. I assume I shouldn't get this double raw + processed logs - that is strange, I have to investigate, it didn't do that for me.
2. We need some sort of help text here explaining what these things are... what does 'processed mean'? I go to it, its empty. I guessed maybe I need to mark something as processed (like reviewed)? but I open a log and I don't see such a button, nor in dropbutton, so I hit a dead end there. - Good catch. These are 2 separate issues:
   1. We need help text. Processed logs are logs which have been processed by LLM and have structured output (nice UX). Now, the only ones we do proces that way, are Chat logs, but that list shall grow. You should start getting something there as soon as you log first call of type Chat.
   2. The tab with Processed logs should have sub-tabs with types of processed logs (again, now just Chat). But you should at least see an empty View there!
3. Is tabs the right interface? The typical interface for this would be Local Tasks I believe, like attached. In which case 'Delete' could also be a local task like it is for Drupal Core database logs rather than the more scary danger button? - The UX here is quite complex. We need:
   • Menu for selecting 'Raw logs' (what you have now) and 'Processed logs' (nice logs, processed by AI, with call chain and all the attributes). With sumbenu for each type of processed logs, because they can have very different fields and post-processing:
     • Chat processed logs
     • Image generation processed logs
     • ...
   • Delete butto - to give the Delete functionality the same prominence like have 'Raw logs' and 'Processed logs' didn't feel right with me, but I might be the only one?

All the points were incorporated. Main changes:

1. Fields now created programmatically at AI Log
2. Added Raw request tracking
3. Menu was changed into tabs, tabs are created via deriver
4. Delete button is now tab
5. LLM processing is async, both via Cron and Batch
6. Tabs were renamed for better clarity

Okay gave this a play! Seems like good improvement!

I'm not sure I am using it heavily enough so maybe needs someone who is e.g. using AI Chatbot with RAG for example to test this out.

When I used with custom rag implementation had to do a bit to get it to log as well due to the conditions in the View, but I think no change there, that's expected. When I used with AI API Explorer, I found a couple things:

1. I received 2 entries in the log for 1 request in 'Chat Generation Explorer'
2. If the request and response are small, the 'Summary' actually expanded on the response instead of summarizing it (openai 4o-mini). I wonder if there needs to be a threshold (strlen?) maybe where no summarize task is done?
3. LLM response in view log for me was markdown, so hard to read in that layout, but maybe supporting that can be a follow-up request and not part of this
4. (nitpick and not necessary) Not a big thing, but it seems like Queue + Process buttons are sub-tasks 'Processed logs' which could possibly be Local Action Links → instead to avoid any disconnect between the list & what needs to be done to populate the list

Beyond that I got fatal error because AssistantStreamIterator::getIterator() extends final method StreamedChatMessageIterator::getIterator() but I assume thats unrelated so I just manually changed to continue testing, but flagging in case.