Week 18 • Day 6Week 18
Day 6:Prometheus Metrics & Kubernetes Health Probes
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 6: Prometheus Metrics & Kubernetes Health Probes (/healthz, /readyz, /metrics)
1. First-Principles: Liveness vs. Readiness in Kubernetes
Kubernetes manages containers using two distinct probe semantics:
Architecture Flow Diagram
Kubernetes Kubelet->Live{1. Liveness Probe /healthz}
|Pass| KeepAlive[Keep Pod Running
|Fail (3x)| Restart[Restart Container (SIGKILL)
Ready{2. Readiness Probe /readyz}
|Pass| RouteTraffic[Route Ingress Traffic to Pod
|Fail| RemoveEndpoints[Temporarily Remove Pod from Load Balancer Endpoints
The Production Disaster of Combining Them:
If an engineer implements only one probe that checks external dependencies (e.g., verifying if the database is reachable):
- When the external database experiences a 10-second transient network blip, every single container in the cluster fails liveness at the same time.
- Kubernetes restarts all 50 pods simultaneously, triggering a massive crashloop thundering herd that takes down the entire application!
- Rule:
/healthzchecks ONLY internal process health (no external calls)./readyzchecks dependency readiness before accepting traffic.
2. Prometheus Metric Types for AI Systems
The /metrics endpoint exposes metrics formatted in standard Prometheus plain text:
- Counter (
llm_requests_total): Monotonically increasing count of total API invocations partitioned byprovider,model, andstatus. - Histogram (
llm_request_duration_seconds): Binned execution latency percentiles to compute p50, p95, and p99 in Grafana. - Gauge (
cache_hit_ratio): Dynamic float value representing the current semantic cache efficiency. - Counter (
llm_tokens_total): Total prompt vs completion tokens billed.
In Simple English: Kubernetes Health Probes (/healthz vs /readyz) & Prometheus
Never mix up these two probes in Kubernetes — mixing them causes cascading cluster outages:
/healthz(Liveness = "Is Python frozen/deadlocked?"): Should be an instant lightweight check returning200 OKif the event loop is alive. NEVER check Redis/LLM API inside/healthz! Otherwise, when Redis blinks for 2 seconds, Kubernetes kills and restarts ALL your healthy AI pods simultaneously!/readyz(Readiness = "Am I ready for user traffic right now?"): Checks DB & cache connections. If Vector DB isn't warm yet, Kubernetes safely holds traffic out of that pod until ready without killing it!