Without streaming, a chat app sends your question and shows a spinner until the model has finished the entire answer, which can take many seconds. With streaming, the first words appear almost immediately and the rest fills in as the model writes. The total generation time is about the same; what changes is how long the user waits to see anything, measured as time to first token.
Most model APIs stream over server-sent events, a web standard for a one-way stream of events from server to browser (WHATWG). With Anthropic’s API, for example, a request with "stream": true returns a sequence of events: a start event, many small text deltas and a stop event (Anthropic, Streaming messages). Voice and real-time apps often use WebSockets instead, so both sides can send at once.
Streaming changes the engineering, not just the look. Code has to assemble partial pieces, handle a connection that drops halfway, and wait for the end before parsing structured output or running a final safety check, because the start of an answer can look fine and the end may not. In episode 6 at 45:00, Twilio’s Vinnie Giarrusso describes this shift for speech-to-speech: moving from simple request-and-response calls to “setting up a WebSocket connection and streaming and just waiting for streams to come back,” which raises new questions about how to evaluate an interaction, including whether the system caught an interruption mid-stream. Episode 6 includes a Galileo sponsor read at 27:00.
Editorially, stream anything a person waits on, and measure both first-token time and total latency, since streaming improves the first and does nothing for the second.
Hear it from the guest
“Setting up a WebSocket connection and streaming and just waiting for streams to come back”
Quotes lightly edited to remove filler words.
Sources
- Anthropic: Streaming messages — Setting "stream": true returns the response incrementally as server-sent events, with message_start, content_block_delta and message_stop events; checked October 24, 2026.
- WHATWG HTML Standard: Server-sent events — The web standard for server-sent events, the one-way HTTP event stream many model APIs use for streaming; checked October 24, 2026.
- Chain of Thought episode 6 at 45:00 — Twilio's Vinnie Giarrusso describes moving from request-response HTTP calls to WebSocket streaming for speech-to-speech and what that means for evals.
