DataArena
Week 22 Master Guide

Week 22: Self-Hosting Open Models (`vLLM`, `Ollama`, GPU Math & Quantization)

Week 22 Daily Breakdown (7 Days)

Day 1

Day 1: The Physics & Math of LLM Inference (VRAM & Bandwidth Calculators)

1 code file(s) • 3 Q&A
Day 2

Day 2: High-Throughput Serving with `vLLM` & PagedAttention

1 code file(s) • 3 Q&A
Day 3

Day 3: Local & Edge Model Serving with `Ollama` & `llama.cpp`

1 code file(s) • 3 Q&A
Day 4

Day 4: GPU Quantization Formats Compared — AWQ, GPTQ, GGUF, FP8

1 code file(s) • 3 Q&A
Day 5

Day 5: Building the Unified OpenAI-Compatible Serving Gateway

1 code file(s) • 3 Q&A
Day 6

Day 6: Automated Cost, Latency & Accuracy Benchmarking

1 code file(s) • 3 Q&A
Day 7

Day 7: Week 22 Mini-Project — High-Throughput Local Serving Engine

1 code file(s) • 3 Q&A

Week 22: Self-Hosting Open Models (vLLM, Ollama, GPU Math & Quantization)

In Week 21, you mastered parameter-efficient fine-tuning (LoRA, QLoRA) and preference alignment (DPO). You trained a custom SRE Incident Classification adapter.

Now comes the critical engineering question: How do you host, serve, and scale these open-weights models in production? Calling hosted proprietary APIs (OpenAI, Anthropic) is simple, but in high-volume production, hosting your own open-weights models (LLaMA-3, Qwen-2.5, Mistral) unlocks 4x to 10x cost reductions, sub-200ms p95 latencies, zero data egress risk, and complete operational independence.

1. The Physics & Roofline Model of LLM Inference

Senior AI Engineers understand that training and inference have completely different hardware bottlenecks:

  • Training & Prefill (Prompt Processing): Highly compute-bound (GEMM: General Matrix Multiply). The arithmetic intensity (FLOPs / byte transferred) is high. Modern GPU Tensor Cores are fully utilized.
  • Decoding (Autoregressive Token Generation): Heavily memory-bandwidth bound (GEMV: General Matrix-Vector Multiply). Every single newly generated token requires streaming every single weight of the model from High Bandwidth Memory (HBM) into SRAM cache!
Architecture Flow Diagram
Phase 1: Prefill (Prompt Processing)<br/>• Computes initial prompt activations<br/>• Parallel GEMM matrix operations<br/>• <b>COMPUTE BOUND</b>
Phase 2: Decode (Token-by-Token)<br/>• Generates 1 token per step<br/>• Matrix-vector multiplications (GEMV)<br/>• <b>MEMORY BANDWIDTH BOUND</b>
Time-to-First-Token (TTFT) determined by Tensor TFLOPs
Inter-Token Latency (ITL) determined by GPU Memory Bandwidth (GB/s)
H1
H2

The Generation Bandwidth Formula

For a single batch stream generating tokens autoregressively on a model of size SmodelS_{\text{model}} (in gigabytes): Generation Latency per Token≈SmodelGPU Memory Bandwidth (GB/s)\text{Generation Latency per Token} \approx \frac{S_{\text{model}}}{\text{GPU Memory Bandwidth (GB/s)}} Max Single-Stream Throughput=Bandwidth (GB/s)Smodel\text{Max Single-Stream Throughput} = \frac{\text{Bandwidth (GB/s)}}{S_{\text{model}}}

Concrete Example (LLaMA-3.1-8B in FP16 = 16 GB):

  • On an NVIDIA RTX 4090 ($1,008\text{ GB/s}$ bandwidth): Max Throughput=1008 GB/s16 GB≈63 tokens/sec\text{Max Throughput} = \frac{1008\text{ GB/s}}{16\text{ GB}} \approx \mathbf{63\text{ tokens/sec}}
  • If quantized to 4-bit (AWQ / GPTQ = 4 GB): Max Throughput=1008 GB/s4 GB≈252 tokens/sec(4× speedup!)\text{Max Throughput} = \frac{1008\text{ GB/s}}{4\text{ GB}} \approx \mathbf{252\text{ tokens/sec}} \quad (\mathbf{4\times \text{ speedup!}}) Quantization does not just save VRAM—it directly multiplies token decoding speed!

2. High-Throughput Serving: PagedAttention & Continuous Batching

In 2023, Woosuk Kwon et al. (UC Berkeley) identified the primary bottleneck in production LLM serving: KV Cache Memory Fragmentation.

Architecture Flow Diagram
Pre-allocates maximum sequence length (e.g. 4,096 tokens)
60% to 80% of KV cache memory is wasted (Internal Fragmentation)
Static batching: GPU stalls waiting for slowest sequence
Inspired by OS Virtual Memory: KV cache split into fixed-size Blocks (16 tokens)
Block Table maps logical tokens to non-contiguous physical GPU pages
Memory waste reduced from 70% to < 4%
Continuous Batching: New requests join the batch at each iteration

PagedAttention Innovations

  1. Near-Zero Memory Waste (<4%< 4\%): Blocks are allocated on demand as tokens are generated.
  2. Continuous Batching (Iteration-Level Scheduling): Finished requests exit immediately; new requests enter at the next token step. The GPU runs at 100% capacity continuously.
  3. Prefix Caching: Identical system prompts and few-shot examples share the same physical KV blocks, dropping Time-To-First-Token (TTFT) by $90%$.

3. Quantization Landscape Compared

FormatQuantization MethodPrimary TargetStrengthsIdeal Production Use Case
AWQActivation-Aware (Protects salient 1% weights)NVIDIA GPUsNear-zero perplexity degradation, ultra-fast CUDA kernels.vLLM production GPU serving (A100, H100, L4).
GPTQSecond-order Hessian error minimizationNVIDIA GPUsVery fast quantization, broad support.Single-GPU inference.
GGUFk-quants (Q4_K_M, Q5_K_M, Q8_0)CPU / Apple Silicon / MetalHighly portable single-file binary format, zero-copy mmap.Ollama / Local dev & edge deployment.
FP8Native 8-bit Float (E4M3 / E5M2)Ada / Hopper (L4, H100)Native Tensor Core hardware execution, no dequantization overhead.Enterprise H100 / L4 clusters.

4. Week 22 Day-by-Day Syllabus

DayTopicSource ModuleAutomated Test
Day 1The Physics & Math of LLM Inference (VRAM & Bandwidth)src/inference_physics_and_vram_calculator.pytests/test_inference_physics_and_vram_calculator.py
Day 2High-Throughput Production Serving with vLLM & PagedAttentionsrc/paged_attention_simulator.pytests/test_paged_attention_simulator.py
Day 3Local & Edge Model Serving with Ollama & llama.cpp (GGUF)src/ollama_model_manager.pytests/test_ollama_model_manager.py
Day 4GPU Quantization Formats Compared: AWQ, GPTQ, GGUF, FP8src/quantization_benchmark_analyzer.pytests/test_quantization_benchmark_analyzer.py
Day 5Building the Unified OpenAI-Compatible Serving Gatewaysrc/openai_compatible_gateway.pytests/test_openai_compatible_gateway.py
Day 6Automated Cost, Latency & Accuracy Benchmarkingsrc/automated_benchmark_suite.pytests/test_automated_benchmark_suite.py
Day 7Mini-Project: High-Throughput Local Model Serving Enginesrc/high_throughput_serving_service.pytests/test_high_throughput_serving_service.py
Week 22: Week 22: Self-Hosting Open Models (`vLLM`, `Ollama`, GPU Math & Quantization) | 24-Week AI Engineer Academy · DataArena