Week 22 • Day 5Week 22
Day 5:Building the Unified OpenAI-Compatible Serving Gateway
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 5: Building the Unified OpenAI-Compatible Serving Gateway
1. The Power of Universal API Compatibility
In production AI architectures, client applications (web frontends, CLI tools, multi-agent orchestrators) should never be tightly coupled to proprietary vendor SDKs.
By implementing the industry-standard OpenAI REST API specification (/v1/chat/completions and /v1/models), your self-hosted open-model serving cluster becomes a drop-in replacement for hosted APIs:
python
# Any standard OpenAI SDK client works out of the box!
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1", # Our self-hosted gateway!
api_key="internal-cluster-key"
)
response = client.chat.completions.create(
model="llama3.1:8b-instruct",
messages=[{"role": "user", "content": "Triage alert: DB pool exhausted"}]
)
2. API Contract Specification
Architecture Flow Diagram
Client as Client / Chat Frontend
GW as FastAPI OpenAI Gateway
Eng as Self-Hosted Model Engine (vLLM / LoRA
>GW: POST /v1/chat/completions {model, messages, stream: true}
>Eng: Allocate KV Cache & Step Decode Loop
Token Streaming (SSE
>GW: Emits token delta
>Client: data: {id: chatcmpl-1, object: chat.completion.chunk, choices: [{delta: {content: token}}]}\n\n
>GW: Sequence finished (finish_reason: stop
>Client: data: [DONE]\n\n
1. Non-Streaming JSON Contract (stream: false)
json
{
"id": "chatcmpl-a93f2",
"object": "chat.completion",
"created": 1729000000,
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Severity: SEV-1. Run: kubectl scale deployment pgpool."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 42,
"completion_tokens": 18,
"total_tokens": 60
}
}
2. Streaming Server-Sent Events Contract (stream: true)
- HTTP Header:
Content-Type: text/event-stream; charset=utf-8 - Each token emitted as:
data: {"id":"chatcmpl-chunk","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Severity"},"finish_reason":null}]}\n\n - Final termination line:
data: [DONE]\n\n
In Simple English: Why Every Open Model Server Speaks the OpenAI JSON Standard
Notice why vLLM, Ollama, LiteLLM, and Together AI all expose /v1/chat/completions and accept the exact same JSON fields (model, messages, stream, temperature):
- Building your backend against this universal OpenAI-compatible contract means you can test locally on free Ollama, scale on managed GPT-4o / Gemini, or flip over to your own self-hosted vLLM GPU cluster by changing 1 environment variable (
BASE_URL) — zero application rewrite!