Improve prompt caching

iannuzzelli commented

2026-07-22 10:16:47 +00:00

(Migrated from gitlab.com)

Prompt caching is invalidated on nearly every request — RAG context is bundled into the cached block

Current behavior

webapp/agent.py:452-460 builds one system-prompt string (system_text) by concatenating:
today's date + the static system.md prompt + optional per-session agent_guidance + writing-mode
instructions + the RAG-retrieved context for this specific question (Documenti rilevanti recuperati dal database:\n\n{context}, appended at line 436-437).

For Anthropic models, this whole concatenated string is sent as a single system block with one
cache_control: {"type": "ephemeral"} breakpoint (line 456-459).

Problem

Prompt caching is a prefix match — a single byte difference anywhere in the cached block
invalidates the whole thing. Since the RAG context (different retrieved chunks for nearly every
question) is folded into the same string as the static instructions, the cache effectively
never hits across turns or across users: every new question means a new set of retrieved docs,
which means a new string, which means a fresh (full-price, ~1.25x write-premium) cache write
instead of a cheap (~0.1x) cache read.

In practice this means caching only pays off within a single turn's multi-iteration agent loop
(when the same RAG context gets re-sent across 2-3 LLM calls before the next user message) — it
never benefits the much larger opportunity of sharing the static instructions across every
request from every user, which don't need to vary at all.

Secondary, lower-impact issue: today's date is also baked into the cached block, so even the
static portion invalidates at least once a day regardless of the RAG issue.

Proposed fix

Anthropic's system field accepts multiple text blocks — split the single string into separate
blocks ordered stable → volatile, with the cache breakpoint only on the stable portion:

system = [
    {"type": "text", "text": STATIC_SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}},
    # per-session guidance / writing mode here, own block, no breakpoint needed unless large
    {"type": "text", "text": f"Documenti rilevanti recuperati dal database:\n\n{context}"},  # no cache_control — changes every turn, expected
]

The static prompt block becomes byte-identical across every request from every user (staff and, once shipped, widget traffic) — not just within one session — so it can actually achieve a high cache-hit rate instead of essentially never hitting.
The RAG context stays uncached (correctly — it's supposed to vary), but it's no longer dragging the static instructions down with it.
Move the date stamp out of the cached block too (either into its own small trailing block, or into the first user-turn message) so the static block doesn't invalidate daily either.
Also worth checking while in this code
MCP tool definitions render before system in the request (tools → system → messages),
and changing the tool set invalidates the tools+system+messages cache tiers together. Confirm the
tool set is fixed for the lifetime of a session (set once from the classified intent) rather than
changing mid-session — if it can change, that's another silent invalidator worth flagging
separately.

Expected impact
Directional, not measured: this should meaningfully raise the cache hit rate on the static portion
of every request, since that portion would go from "essentially never cached" to "cached across
the entire app's traffic." Actual $ savings depend on the static-prompt-to-RAG-context size ratio
— worth measuring via response.usage.cache_read_input_tokens before/after rather than assuming
a specific percentage.

## Prompt caching is invalidated on nearly every request — RAG context is bundled into the cached block ### Current behavior `webapp/agent.py:452-460` builds one system-prompt string (`system_text`) by concatenating: today's date + the static `system.md` prompt + optional per-session `agent_guidance` + writing-mode instructions + **the RAG-retrieved context for this specific question** (`Documenti rilevanti recuperati dal database:\n\n{context}`, appended at line 436-437). For Anthropic models, this whole concatenated string is sent as a single system block with one `cache_control: {"type": "ephemeral"}` breakpoint (line 456-459). ### Problem Prompt caching is a **prefix match** — a single byte difference anywhere in the cached block invalidates the whole thing. Since the RAG context (different retrieved chunks for nearly every question) is folded into the *same* string as the static instructions, the cache effectively never hits across turns or across users: every new question means a new set of retrieved docs, which means a new string, which means a fresh (full-price, ~1.25x write-premium) cache write instead of a cheap (~0.1x) cache read. In practice this means caching only pays off *within* a single turn's multi-iteration agent loop (when the same RAG context gets re-sent across 2-3 LLM calls before the next user message) — it never benefits the much larger opportunity of sharing the **static instructions** across every request from every user, which don't need to vary at all. Secondary, lower-impact issue: today's date is also baked into the cached block, so even the static portion invalidates at least once a day regardless of the RAG issue. ### Proposed fix Anthropic's `system` field accepts multiple text blocks — split the single string into separate blocks ordered stable → volatile, with the cache breakpoint only on the stable portion: ```python system = [ {"type": "text", "text": STATIC_SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}, # per-session guidance / writing mode here, own block, no breakpoint needed unless large {"type": "text", "text": f"Documenti rilevanti recuperati dal database:\n\n{context}"}, # no cache_control — changes every turn, expected ] ``` The static prompt block becomes byte-identical across every request from every user (staff and, once shipped, widget traffic) — not just within one session — so it can actually achieve a high cache-hit rate instead of essentially never hitting. The RAG context stays uncached (correctly — it's supposed to vary), but it's no longer dragging the static instructions down with it. Move the date stamp out of the cached block too (either into its own small trailing block, or into the first user-turn message) so the static block doesn't invalidate daily either. Also worth checking while in this code MCP tool definitions render before system in the request (tools → system → messages), and changing the tool set invalidates the tools+system+messages cache tiers together. Confirm the tool set is fixed for the lifetime of a session (set once from the classified intent) rather than changing mid-session — if it can change, that's another silent invalidator worth flagging separately. Expected impact Directional, not measured: this should meaningfully raise the cache hit rate on the static portion of every request, since that portion would go from "essentially never cached" to "cached across the entire app's traffic." Actual $ savings depend on the static-prompt-to-RAG-context size ratio — worth measuring via response.usage.cache_read_input_tokens before/after rather than assuming a specific percentage.

iannuzzelli commented

2026-07-22 10:16:57 +00:00

(Migrated from gitlab.com)

changed the description

iannuzzelli commented

2026-07-22 10:17:37 +00:00

(Migrated from gitlab.com)

changed the description

iannuzzelli commented

2026-07-22 10:17:48 +00:00

(Migrated from gitlab.com)

assigned to @iannuzzelli