AIOrchestrator framework and model prefix caching

Flow
1

More findings from further testing.

Environment

AWS Bedrock via Spring AI 2.0.1
Model: Amazon Nova Pro

Nova Pro supports up to 4 cache checkpoints per request, with a total of 20k tokens, 5 minute TTL.
Nova Pro does NOT support tool caching.

(Findings fleshed out and organized by Opus 5)

1. withMetadata() content lands in a TOOL DESCRIPTION — the worst place for caching

AIOrchestrator.mergeWithContextTool() wraps the metadata supplier’s result in a synthetic get_session_context tool and carries the content in that tool’s description .

Three consequences:

  1. Tool definitions precede system in Bedrock’s cache hierarchy, so per-turn metadata sits in the cache prefix. Any system checkpoint is invalidated on every request where the metadata string differs.
  2. Nova cannot checkpoint tools at all, so this content is not independently cacheable either.
  3. The default supplier (defaultContextSupplier()) renders the server clock to the minute: "Current server date and time: 2026-05-28T17:42+03:00 (Friday, Europe/Helsinki)". Out of the box, therefore, Vaadin invalidates system prompt caching every 60 seconds — shorter than the 5-minute TTL, so a low-traffic app may never see a cache hit.

Suggestion: deliver per-turn metadata as a trailing system block or prepended to the user message, not as a tool description. At minimum, document the caching cost and make the clock-in-metadata default opt-in.

2. The system prompt is fixed at build time, so per-turn stable context cannot be cached

AIOrchestrator stores one immutable String systemPrompt and reuses it for every turn. Context that is stable for a request but chosen per request — our routed schema segments are the exact case — has nowhere cacheable to go. It must be returned from a tool, which lands in messages after the variable user question, i.e. the least cacheable position available.

Suggestion: accept additional content for the system prompt, resolved per turn, so applications can put per-turn-but-stable context in a cacheable channel.

3. A single system String defeats Spring AI’s multi-block caching — silently

Spring AI 2.0.0 has BedrockCacheOptions.multiBlockSystemCaching , which places the checkpoint after the second-to-last system block so a static prefix caches while a dynamic tail changes freely. That is exactly the two-level structure we want, and it requires two or more SystemMessage s.

SpringAILLMProvider calls promptSpec.system(oneString) , producing exactly one. In BedrockProxyChatModel:

int cacheBoundaryIndex = multiBlockSystemCaching ? systemMessageList.size() - 2
                                                 : systemMessageList.size() - 1;

With one system message and multiBlockSystemCaching enabled the index is -1 and no cache point is emitted at all — caching is silently disabled rather than degraded. A trap for anyone who turns the flag on while using Vaadin’s provider.

Suggestion (Vaadin): support multiple system blocks. Suggestion (Spring AI): guard the degenerate case — warn, or fall back to size()-1 , when multiBlockSystemCaching is set with fewer than two system messages.

4. LLMRequest has no caching hook

The interface exposes userMessage , systemPrompt , tools , explicitTools , attachments — no way to express “cache up to here”. We configured caching on the ChatModel bean’s default options instead, which works but is provider-specific and invisible to the orchestrator.

Suggestion: a provider-agnostic caching hint on LLMRequest / the builder.