I know this is not specifically a Vaadin topic (gotta say it before Simon yells at me :) but this is my favorite forum with a lot of helpful developers and I thought it would be more fun to discuss here than in @marcushellberg 's GitHub.
There is a BIG problem in Spring AI (and probably langchain4j) that no one is talking about. Tool calls are not stored in chat memory. If you add tool calling and chat memory to the same project, it will fail. The first command will use tool calling, but then the next one will not because the LLM sees that the previous messages were successful without it.
See Marcus’s example and note that he does not use chat memory. Marcus, maybe you are aware of this issue and have developed your own solution?
For me, this makes Spring AI almost completely unusable. Chat memory is required for a good user experience.
What workarounds have to tried to solve this problem? Have you implemented your own custom chat memory to try to include the tool calls?
One thing I’ve noticed which may be related that SimpleLoggingAdvisor doesn’t seem to be logging the tool calls. The actual tool invocations print no logs and the called tools don’t show up in the model request logs ( "toolCalls" : [ ]). The tool calls do seem to be invoked properly - if you add a log statement in the tool method, it gets printed ok, and the conversation remembers the called tools if you ask about them (like in your example).
Yes. There are several similar issues. I have already commented on one recently. This has been an issue for over a year so it’s hard to tell if there will be a fix any time soon.
I set out to test this more thoroughly. I was not able to prove my initial claim that chat memory breaks tool calling, but I did find a cheaper way to do it.
