Week 22 • Day 7Week 22
Day 7:Week 22 Mini-Project — High-Throughput Local Serving Engine
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 7: Week 22 Mini-Project — High-Throughput Local Serving Engine
1. Project Overview & Architecture
In this mini-project, we synthesize all concepts from Week 22 into a Production High-Throughput Local Model Serving Engine:
- PagedAttention KV-Cache Allocator: Manages physical GPU block frames (16-token pages) with Automatic Prefix Caching (APC) to eliminate redundant prefill.
- Iteration-Level Continuous Batcher: Preempts, admits, and evicts sequences dynamically at single-token granularity to maximize GPU saturation.
- OpenAI-Compatible REST Gateway: Exposes standard
/v1/chat/completions(JSON & SSE streaming) and/v1/models. - Real-Time Telemetry & Prometheus
/metrics: Tracks active physical pages, KV-cache fragmentation percentage, TTFT, and decoding tokens-per-second.
Architecture Flow Diagram
OpenAI Python SDK Client
Web Chatbot Streaming UI
EP[FastAPI OpenAI Gateway: /v1/chat/completions
EP
CBS[Continuous Batch Scheduler (Iteration-Level)
APC[Prefix Caching Engine (Hash Lookup)
PMM[PagedAttention Memory Manager (16-token pages)
DECODE[Autoregressive Decode Step (AWQ / FP8 GEMV)
SSE[SSE Streamer (data: {delta...}\n\n)
METRICS[GET /metrics (Prometheus Format)
2. Key Production Verification Metrics
| Capability | Target SLA | Production Verification |
|---|---|---|
| API Contract Parity | 100% OpenAI Spec | Validated with standard OpenAI payloads & streaming chunks. |
| Memory Fragmentation | internal waste | Enforced by 16-token page granularity vs contiguous allocation. |
| Prefix Cache Hit Rate | on shared prompt | Zero re-prefill on repeated system runbooks. |
| Decode Latency | per token | High-speed memory bandwidth streaming. |
| Observability | Real-time Prometheus metrics | Exported on /metrics endpoint. |
In Simple English: Putting Self-Hosting Together (Week 22 Mini-Project)
In Tab 2 (Codes) & Tab 4 (Test & Troubleshooting) you assemble a production local serving layer:
- Calculates exact VRAM & KV-Cache fit before picking a model + quantization (
Q4_K_M/AWQ/FP8). - Serves traffic behind an OpenAI-compatible
/v1/chat/completionscontract supporting streaming deltas cleanly. - Benchmarks
TTFT,TPS, and cost per 1,000 chats vs frontier cloud APIs!