Chat responses aren't actually streamed, full answer arrives in one shot #12

New issue

Closed

opened 2026-09-11 14:56:10 +00:00 by francesco · 0 comments

francesco commented

2026-09-11 14:56:10 +00:00

Owner

Summary

Both the public chat widget and the internal webapp show a "…" placeholder for
the entire duration of the final answer generation, then the complete response
appears all at once. There is no real token-by-token streaming anywhere in the
current agent implementation, despite the transport (SSE) and the widget's
client-side code already being built to support it.

Observed behavior

Tested locally via the public widget (/chat), question: "Quali posizioni ha
PeaceLink sulla deterrenza nucleare?"

Total request latency: ~28s (confirmed via Langfuse trace).
Breakdown: RAG retrieval 8.15s, tool-routing LLM call 2.5s, tool execution
0.5s, final-answer LLM call 24.8s (1,179 output tokens, ≈47 tok/s — a
normal generation rate for Claude Sonnet on a long response, not a hang).
The widget shows the "…" placeholder for the entire 24.8s, then the full
~4,000-character answer appears at once.

Root cause

POST /chat (ai/assistant/api/main.py) streams an SSE response and already
plumbs a partial_callback all the way through to the agent
(ai/assistant/webapp/agent.py, _async_run → _run_graph), so the
transport layer is not the problem.

The problem is in _run_graph's event loop
(ai/assistant/webapp/agent.py:293-358), which consumes
graph.astream_events(...) (LangGraph's event stream) and only calls
partial_callback here:

if name == "agent" and partial_callback:
    out = event.get("data", {}).get("output")
    if isinstance(out, dict):
        msgs = out.get("messages", [])
        if msgs:
            last = msgs[-1]
            content = ...
            tool_calls = ...
            if content and isinstance(content, str) and content.strip() and tool_calls \
                    and "sanitised" not in content and "sanitized" not in content:
                partial_callback(content)

Two issues:

This fires on on_chain_end for the "agent" node — i.e. once an entire
LangGraph node (one full, already-completed LLM call) finishes. It is not
token-level streaming; it's a coarse "a step just finished" relay.
The condition requires both content and tool_calls to be present.
The final answer node — the one that actually writes the response the user
reads — by definition has no tool_calls (there's nothing left to call).
So this condition is never true for the final answer, and
partial_callback is never invoked for it at all.

The only content the client ever receives for the final answer is the single
done event, sent from api/main.py after agent._async_run(...) returns —
i.e. after the entire LLM generation has already finished server-side.

The widget's own code (ui/js/chat-widget.js, readSseStream) already
handles this correctly on the receiving end — it renders payload.text on
every partial event and would show growing text if the events existed. The
gap is entirely server-side.

Impact

Public widget (api/main.py → shared agent.py) and internal webapp
(webapp/chat.py → same agent.py) are both affected — this is shared
code, not a widget-only issue.
Any response that doesn't end in a tool call (i.e. most final answers) waits
in full before anything renders. Longer answers feel proportionally worse
since there's no feedback that anything is happening beyond the existing
"…" placeholder.

Proposed fix

Switch the LLM call inside the "agent" LangGraph node from a blocking call to
a streaming one (e.g. LiteLLM/LangChain .astream() instead of the current
invoke-style call), and forward each chunk through partial_callback as it
arrives, instead of only once at node-end.

Scope: this means changing the agent's node implementation in
webapp/agent.py (shared by both consumers), not just the API/widget layer.
Both api/main.py's SSE relay and the widget's readSseStream already
support incremental partial events, so no changes should be needed on
either of those once the agent itself actually streams.

Not investigated yet

Exactly where in agent.py the "agent" node's LLM call is made (separate
from the _run_graph event-loop code referenced above) — needed to know
how invasive the switch to streaming is.
Whether LangGraph's astream_events naturally surfaces
on_chat_model_stream (or similar per-token) events already, which might
make this simpler than manually switching the underlying LLM call — worth
checking before assuming a bigger rewrite is needed.
Any interaction with the rolling-summary / Langfuse generation-tracking code
that currently assumes a single complete content string per iteration.

## Summary Both the public chat widget and the internal webapp show a "…" placeholder for the entire duration of the final answer generation, then the complete response appears all at once. There is no real token-by-token streaming anywhere in the current agent implementation, despite the transport (SSE) and the widget's client-side code already being built to support it. ## Observed behavior Tested locally via the public widget (`/chat`), question: "Quali posizioni ha PeaceLink sulla deterrenza nucleare?" - Total request latency: ~28s (confirmed via Langfuse trace). - Breakdown: RAG retrieval 8.15s, tool-routing LLM call 2.5s, tool execution 0.5s, **final-answer LLM call 24.8s** (1,179 output tokens, ≈47 tok/s — a normal generation rate for Claude Sonnet on a long response, not a hang). - The widget shows the "…" placeholder for the entire 24.8s, then the full ~4,000-character answer appears at once. ## Root cause `POST /chat` (`ai/assistant/api/main.py`) streams an SSE response and already plumbs a `partial_callback` all the way through to the agent (`ai/assistant/webapp/agent.py`, `_async_run` → `_run_graph`), so the transport layer is not the problem. The problem is in `_run_graph`'s event loop (`ai/assistant/webapp/agent.py:293-358`), which consumes `graph.astream_events(...)` (LangGraph's event stream) and only calls `partial_callback` here: ```python if name == "agent" and partial_callback: out = event.get("data", {}).get("output") if isinstance(out, dict): msgs = out.get("messages", []) if msgs: last = msgs[-1] content = ... tool_calls = ... if content and isinstance(content, str) and content.strip() and tool_calls \ and "sanitised" not in content and "sanitized" not in content: partial_callback(content) ``` Two issues: 1. This fires on `on_chain_end` for the `"agent"` node — i.e. once an entire LangGraph node (one full, already-completed LLM call) finishes. It is not token-level streaming; it's a coarse "a step just finished" relay. 2. The condition requires **both** `content` and `tool_calls` to be present. The final answer node — the one that actually writes the response the user reads — by definition has no `tool_calls` (there's nothing left to call). So this condition is never true for the final answer, and `partial_callback` is never invoked for it at all. The only content the client ever receives for the final answer is the single `done` event, sent from `api/main.py` after `agent._async_run(...)` returns — i.e. after the entire LLM generation has already finished server-side. The widget's own code (`ui/js/chat-widget.js`, `readSseStream`) already handles this correctly on the receiving end — it renders `payload.text` on every `partial` event and would show growing text if the events existed. The gap is entirely server-side. ## Impact - Public widget (`api/main.py` → shared `agent.py`) and internal webapp (`webapp/chat.py` → same `agent.py`) are both affected — this is shared code, not a widget-only issue. - Any response that doesn't end in a tool call (i.e. most final answers) waits in full before anything renders. Longer answers feel proportionally worse since there's no feedback that anything is happening beyond the existing "…" placeholder. ## Proposed fix Switch the LLM call inside the "agent" LangGraph node from a blocking call to a streaming one (e.g. LiteLLM/LangChain `.astream()` instead of the current invoke-style call), and forward each chunk through `partial_callback` as it arrives, instead of only once at node-end. Scope: this means changing the agent's node implementation in `webapp/agent.py` (shared by both consumers), not just the API/widget layer. Both `api/main.py`'s SSE relay and the widget's `readSseStream` already support incremental `partial` events, so no changes should be needed on either of those once the agent itself actually streams. ## Not investigated yet - Exactly where in `agent.py` the "agent" node's LLM call is made (separate from the `_run_graph` event-loop code referenced above) — needed to know how invasive the switch to streaming is. - Whether LangGraph's `astream_events` naturally surfaces `on_chat_model_stream` (or similar per-token) events already, which might make this simpler than manually switching the underlying LLM call — worth checking before assuming a bigger rewrite is needed. - Any interaction with the rolling-summary / Langfuse generation-tracking code that currently assumes a single complete `content` string per iteration.

francesco changed title from ~~Chat responses aren't actually streamed — full answer arrives in one shot~~ to Chat responses aren't actually streamed, full answer arrives in one shot

2026-09-11 14:56:20 +00:00

francesco added this to the AI Assistant project

2026-09-11 14:56:26 +00:00

francesco self-assigned this

2026-09-11 14:56:28 +00:00

francesco added the

P2: important

label

2026-09-11 14:56:34 +00:00

francesco referenced this issue from a commit

2026-09-11 16:00:43 +00:00

#12 Stream responses