Month 6 Capstone: Fine-Tuned Open Model Serving & Benchmark Platform

Production AI Engineering Curriculum — Capstone Project 6 of 6

1. Executive Summary & Business Objective

Monolithic proprietary foundation models (OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet) are powerful generalists, but enterprise operations face severe pain points:

Excessive Recurring OpEx: High-volume workloads (10M+ queries/month) cost tens of thousands of dollars monthly in token egress fees.
Network & Decoding Latency: Hosted round-trip latency often exceeds $800\text{ ms} $to \$ 1,500\text{ ms}$ for short classification outputs.
Data Sovereignty & Security Mandates: Sending internal operational telemetry, customer PII, or proprietary logs to public cloud endpoints violates HIPAA, PCI-DSS, and SOC2 policies.

Capstone 6 solves this by establishing a complete, self-hosted open-weights AI infrastructure:

Task-Specific Fine-Tuning (src/train_lora.py): Fine-tunes an open foundation model (LLaMA-3 / Qwen-2.5) using Parameter-Efficient Low-Rank Adaptation (LoRA $r=16, \alpha=32$ ) on domain incident logs.
High-Throughput Serving Engine (src/serving.py): Serves the adapted model with PagedAttention (16-token page granularity, zero-fragmentation KV caches, Continuous Batching) exposing an OpenAI-compatible drop-in REST API (/v1/chat/completions).
Automated Audit Benchmark Suite (src/evaluate.py): Mathematically proves $> 4\times$ cost reduction and $> 3\times$ latency speedup while matching hosted model accuracy ( $> 92\%$ ).

Architecture Flow Diagram

Raw Incident Alert: PostgreSQL connection pool exhausted timeout

1. LoRA Trainer (train_lora.py) • ChatML turn formatting • Assistant-only loss masking • Zero-latency weight merging

2. Production Serving (serving.py) • PagedAttention KV-Cache (16-token blocks) • Continuous Batch Scheduler • OpenAI /v1/chat/completions & /metrics

3. Benchmark Suite (evaluate.py) • TTFT & Inter-Token Latency • p50 / p95 / p99 Percentiles • Cost Arbitrage Audit

SERVE->EVAL

Severity: SEV-1 | Domain: DATABASE

Mitigation Runbook: pg_terminate_backend() & pool scale-out

Cost Arbitrage: 5.6x cheaper than hosted API

SEV & CMD & SAVINGS

Capstone6_Engine

2. Directory Structure

text

capstones/month06_finetuned_open_model_engine/
├── README.md                          # Executive architecture documentation
├── Dockerfile                         # Production multi-stage hardened container
├── docker-compose.yml                 # Standalone service orchestration
├── src/
│   ├── train_lora.py                  # LoRA parameter-efficient training pipeline
│   ├── serving.py                     # vLLM-style PagedAttention serving microservice
│   └── evaluate.py                    # Automated cost, latency & accuracy benchmark runner
└── tests/
    └── test_capstone_month06.py       # End-to-end integration and verification suite

3. Mathematical Principles & Formulas

1. LoRA Matrix Decomposition

Frozen base weights $W_0 \in \mathbb{R}^{d \times k}$ are updated via rank $r \ll \min(d, k)$ : $W = W_0 + \frac{\alpha}{r} B A$ where $B \in \mathbb{R}^{d \times r} = \mathbf{0}$ and $A \in \mathbb{R}^{r \times k} \sim \mathcal{N}(0, \sigma^2)$ . Zero-latency production inference collapses the matrices: $W_{\text{merged}} = W_0 + \frac{\alpha}{r} B A$

2. Memory Bandwidth Roofline Decode Throughput

Autoregressive token generation speed is bottlenecked by GPU High Bandwidth Memory: $\text{Tokens Per Second} \approx \frac{\text{GPU Bandwidth (GB/s)}}{\text{Model Size (GB)}}$ Quantizing an 8B model from 16-bit ($16\text{ GB} $) to 4-bit AWQ (\$ 4\text{ GB}$) quadruples decoding throughput on identical hardware.

3. PagedAttention Memory Allocation

By translating logical sequence blocks to non-contiguous 16-token physical GPU pages, internal and external memory fragmentation is reduced from $> 70\%$ to $< 4\%$ .

4. Running and Verifying Capstone 6

bash

# 1. Run complete automated unit test suite
python -m pytest capstones/month06_finetuned_open_model_engine/tests -v

# 2. Run LoRA training pipeline
python capstones/month06_finetuned_open_model_engine/src/train_lora.py

# 3. Launch serving microservice
python -m uvicorn capstones.month06_finetuned_open_model_engine.src.serving:app --host 0.0.0.0 --port 8000

# 4. Run automated cost and latency benchmark
python capstones/month06_finetuned_open_model_engine/src/evaluate.py