Week 17 • Day 2Week 17

Day 2:Real-Time Streaming APIs — Server-Sent Events (SSE) & Token Generators

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 2: Real-Time Streaming APIs — Server-Sent Events (SSE) & Token Generators

1. First-Principles Problem: Perceived Latency & TTFT vs. TTLT

In conversational AI, the human brain perceives latency through Time-to-First-Token (TTFT) rather than Time-to-Last-Token (TTLT):

  • Suppose an LLM generates N=400N = 400 tokens at a rate of R=40 tokens / secondR = 40\text{ tokens / second}.
  • The total generation duration is: TTLT=TTFT+NR=0.4s+40040=10.4 seconds\text{TTLT} = \text{TTFT} + \frac{N}{R} = 0.4\text{s} + \frac{400}{40} = 10.4\text{ seconds}
  • Without Streaming (Unary JSON): The user stares at an empty loading spinner for 10.4 seconds. 78% of users abandon the page if initial response exceeds 3 seconds.
  • With Streaming (SSE): The user sees the first token in 400ms and reads at human reading speed (~5 tokens/second), perceiving the AI as "instantaneous."

2. Server-Sent Events (text/event-stream) Wire Protocol

Unlike WebSockets (which require bidirectional TCP state and custom proxy configurations), Server-Sent Events (SSE) run over standard unidirectional HTTP/1.1 or HTTP/2:

SSE Wire Format (W3C Specification)

http
HTTP/1.1 200 OK Content-Type: text/event-stream Cache-Control: no-cache Connection: keep-alive X-Accel-Buffering: no event: thought data: {"step": "investigating_k8s_pods", "message": "Querying pod logs..."} event: token data: {"delta": "Cluster "} event: token data: {"delta": "latency "} event: citation data: {"title": "SRE Runbook", "uri": "runbook://sre/high-latency"} event: done data: [DONE]

[!IMPORTANT] Always set X-Accel-Buffering: no for Nginx / reverse proxies! Otherwise, Nginx buffers the SSE chunks in 4KB blocks before forwarding to the client, destroying real-time streaming.


3. Disconnection & Orphan Token Prevention

If a user closes their browser tab or clicks "Stop Generating" mid-stream, an unmonitored server will continue generating all remaining 350 tokens, burning API credits and GPU memory. A production streaming generator monitors the ASGI http.disconnect receive channel: If Client Disconnects  ⟹  Break Generator Loop  ⟹  Cancel Upstream LLM Stream\text{If Client Disconnects} \implies \text{Break Generator Loop} \implies \text{Cancel Upstream LLM Stream}


In Simple English: Why Token Streaming (SSE) Changes User Psychology (TTFT)

When an AI takes 10 seconds to write a 400-word essay:

  • Without Streaming (Blocking REST): User stares at a blank loading spinner for all 10.0 seconds — half of them close the browser tab thinking the site crashed!
  • With Server-Sent Events (SSE): The first word arrives in 0.4 seconds (TTFT) and types smoothly across the screen while the rest computes. Users feel instant responsiveness even though total generation time (TTLT) is identical!
  • Disconnect Cancellation: If the user hits Stop / Closes Tab, breaking out of the Python async for loop saves GPU/API dollars on all unwritten tokens!