Extend Logging Capabilities

Created on 11 December 2024, 2 months ago

Problem/Motivation

Enhance the AI Logging module to provide more granular logging capabilities, better filtering, and integration with the new process system.

Current logging system needs enhancement to:

  • Distinguish between LLM and Agent operations
  • Provide different logging levels
  • Support process chain tracking
  • Better integrate with monitoring

Proposed resolution

  1. Enhanced Log Types
    • LLM operations logging
    • Agent operations logging
  2. Logging Levels for LLM operations
    • Error level
    • Info level
  3. UI Improvements
    • Enhanced log viewer
    • Better filtering interface

API changes

  • Expose integration API

Data model changes

  • New log type
🌱 Plan
Status

Active

Version

1.0

Component

Other Submodules

Created by

🇬🇧United Kingdom seogow

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @seogow
  • 🇬🇧United Kingdom seogow
  • 🇬🇧United Kingdom yautja_cetanu

    Ok firstly this is amazingly cool!!!
    So the good things we need to do are:

    • Maybe create a video explaining it.
    • Maybe a blog post showing screenshots of each page and why its useful and how to use it.
    • Would love to bring emma into it and see what she thinks
    • Tried downloading a prompt and used it in playground and it gave roughly the same answer as in drupal. Super cool for figuring out what is going on.
    • Connect this with Evaluations.
    • Have a button on DeepChat (There needs to be a setting to turn this on) that allows you to highlight an assistant message and go to the log directly.

    Couple of points:

    • What happens if people don't want to spend money on API calls to summarise any log. Is there a setting to turn this off and is there a fall back?
    • It would be good to be able to export the log itself and see a similarly well structured log file that is human readable but also machine readable. I'd like it so that multiple front-ends could visualise the logs differently. (For example a react tree of agents, minikanban could replay the logs)
    • I'm personally finding myself lost with the "Child" logs and where the one I'm looking at fits in and relates to the child logs. Don't have an immediate solution for it yet
  • 🇬🇧United Kingdom yautja_cetanu

    Definitely todo:

    Fallback for no LLM summaries.

    - Requested Actions - Goes to> "First few lines of last user prompt"
    - Performaed action- Goes to> "First few lines of that is the body of the response"

    Follow up issues:

    Explore how good smaller cheaper models are
    Explore embedding a small opensource llm to handle this.

  • First commit to issue fork.
  • 🇬🇧United Kingdom justanothermark

    - Merge AI Logging LLM settings into the AI logging settings
    - Allow a setting that turns off AI summarisation of logs
    - If the setting is turned off, put a warning on the report page "This log overview page is significantly improved with AI summaries that can be turned on here"

    Pushed these changes to a new branch 3493277-extend-logging-capabilities-v2

  • Merge request !471Resolve #3493277 "Extend logging capabilities" → (Open) created by seogow
  • 🇩🇪Germany marcus_johansson

    First off, it looks really nice! I added some comments outside of the stuff phpcs etc. found.

    The biggest issue right now seems to be that all other operation types logging are gone? While some of them do not make sense at the moment, moderation and embeddings for isntance probably could do, to find out why something got banned or if something was searched for.

    Also it adds Performed Actions and Last Performed Actions on any chat log, which also makes little sense, even if it doesn't show it?

    Most of my comments are NIT - but I think we need to have a bundle for other operation types in there, otherwise we don't have feature parity?

Production build 0.71.5 2024