Integrate incremental agent loop execution in XB AI

Moving it to needs work, as message loading icon disappear in case of looping.

narendrar → changed the visibility of the branch 3531000-integrate-incremental-agent to hidden.

kunal.sachdev → made their first commit to this issue’s fork.

kunal.sachdev → changed the visibility of the branch 1.x to hidden.

Note that this will most likely be possible to stream in 1.2.0 as well, see ✨ Allow tool calling in streamed chat Active .

Switching to the correct tag

Adding the video to show how it's working currently →

Currently, only messages from executing agents are displayed. I think we should also find a way to display messages from the tools that are executed.

@kunal.sachdev - the issue is two fold here. OpenAI usually do not provide a text and a tool usage in the same response, unless very specifically prompted to do so, and even then it happens in like 25% of cases. Anthropic on the other hand, usually you have to tell to not include it.

What we did in the AI Assistants API was that, if you do not get a text message back we write Calling X tool as the text message, so there is some feedback.

In this case you could even make assumptions on what its trying to do and maybe write something more intuitive.

We have thought about in the AI Assistants API, to ask with a simple prompt "Look at the following request and the following response and explain in one sentence what it will do.". Since the token generation is usually what takes, the extra effort will make the process quicker anyway. But that is just an idea - the thing above should be a good start.

This is how it's working now!!
→

Great to see some progress on this! Some feedback:

1. We should include implementing the designs for this as part of this issue; it looks currently too unpolished. I've attached video of the designs for this.
2. We should never show the user "Calling X tool" or "Calling X agent". We need to always convert these to more user friendly messages.

There are a couple of things we need to make this nice status of a "Plan of tools"

• One issue is the pure UI as discussed (whether or not we use the word tool, etc).
• The other issue is, can we make it so that AI creates a plan? (Not a fake plan in the prompt) (can look at minikanban)
• The other issue is, what happens if the plan breaks halfway through (it tries to create something that already exists?) do we update the plan?

Might these are 3 seperate issues.

Next steps:

- It would be good to come up with a prompt which will produce a plan we think is good, to see if we can make AI write up the plan and see if we can make it execute.
- 1 example (Go through all 100 pieces of content and check if it says this inccorect fact?)

See also #3533079-4: Introduce AI Agents and tools to create entire page templates using available component entities → for a potential incremental improvement from #20 that isn't quite as nice as #21

I worked on the feature allowing the AI to generate an execution plan, which is then displayed on the screen. The next step is to figure out how to check off each item in the execution plan as the corresponding tool completes its task. The main challenge here is that tool results are only provided once the entire agent called from orchestrator has finished.
For eg - In case of page builder task, AI creates a plan something like:

• Adding components to the page:
• component 1
• component 2
• component 3
• Updating title of the page
• Updating metadata of the page

However, all tool results become available only when the page builder agent completes, so we miss out on intermediary progress updates for each item as they're finished.

I've uploaded the above mp4 as an unlisted Youtube video. Just makes it easier when sharing the designs.

https://www.youtube.com/watch?v=yOy7Ng7U-E4

📌 Allow the Assistant and Chatbot access to the tool calling within the sub-agents behind agent tool calls. Active - Created this related issue.