Day 2:Real-Time Streaming APIs — Server-Sent Events (SSE) & Token Generators
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 2: Real-Time Streaming APIs — Server-Sent Events (SSE) & Token Generators
1. First-Principles Problem: Perceived Latency & TTFT vs. TTLT
In conversational AI, the human brain perceives latency through Time-to-First-Token (TTFT) rather than Time-to-Last-Token (TTLT):
- Suppose an LLM generates tokens at a rate of .
- The total generation duration is:
- Without Streaming (Unary JSON): The user stares at an empty loading spinner for 10.4 seconds. 78% of users abandon the page if initial response exceeds 3 seconds.
- With Streaming (SSE): The user sees the first token in 400ms and reads at human reading speed (~5 tokens/second), perceiving the AI as "instantaneous."
2. Server-Sent Events (text/event-stream) Wire Protocol
Unlike WebSockets (which require bidirectional TCP state and custom proxy configurations), Server-Sent Events (SSE) run over standard unidirectional HTTP/1.1 or HTTP/2:
SSE Wire Format (W3C Specification)
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
X-Accel-Buffering: no
event: thought
data: {"step": "investigating_k8s_pods", "message": "Querying pod logs..."}
event: token
data: {"delta": "Cluster "}
event: token
data: {"delta": "latency "}
event: citation
data: {"title": "SRE Runbook", "uri": "runbook://sre/high-latency"}
event: done
data: [DONE]
[!IMPORTANT] Always set
X-Accel-Buffering: nofor Nginx / reverse proxies! Otherwise, Nginx buffers the SSE chunks in 4KB blocks before forwarding to the client, destroying real-time streaming.
3. Disconnection & Orphan Token Prevention
If a user closes their browser tab or clicks "Stop Generating" mid-stream, an unmonitored server will continue generating all remaining 350 tokens, burning API credits and GPU memory.
A production streaming generator monitors the ASGI http.disconnect receive channel:
In Simple English: Why Token Streaming (SSE) Changes User Psychology (TTFT)
When an AI takes 10 seconds to write a 400-word essay:
- Without Streaming (Blocking REST): User stares at a blank loading spinner for all 10.0 seconds — half of them close the browser tab thinking the site crashed!
- With Server-Sent Events (
SSE): The first word arrives in 0.4 seconds (TTFT) and types smoothly across the screen while the rest computes. Users feel instant responsiveness even though total generation time (TTLT) is identical! - Disconnect Cancellation: If the user hits Stop / Closes Tab, breaking out of the Python
async forloop saves GPU/API dollars on all unwritten tokens!