Week 18 • Day 6Week 18

Day 6:Prometheus Metrics & Kubernetes Health Probes

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 6: Prometheus Metrics & Kubernetes Health Probes (/healthz, /readyz, /metrics)

1. First-Principles: Liveness vs. Readiness in Kubernetes

Kubernetes manages containers using two distinct probe semantics:

Architecture Flow Diagram
Kubernetes Kubelet->Live{1. Liveness Probe /healthz}
|Pass| KeepAlive[Keep Pod Running
|Fail (3x)| Restart[Restart Container (SIGKILL)
Ready{2. Readiness Probe /readyz}
|Pass| RouteTraffic[Route Ingress Traffic to Pod
|Fail| RemoveEndpoints[Temporarily Remove Pod from Load Balancer Endpoints

The Production Disaster of Combining Them:

If an engineer implements only one probe that checks external dependencies (e.g., verifying if the database is reachable):

  • When the external database experiences a 10-second transient network blip, every single container in the cluster fails liveness at the same time.
  • Kubernetes restarts all 50 pods simultaneously, triggering a massive crashloop thundering herd that takes down the entire application!
  • Rule: /healthz checks ONLY internal process health (no external calls). /readyz checks dependency readiness before accepting traffic.

2. Prometheus Metric Types for AI Systems

The /metrics endpoint exposes metrics formatted in standard Prometheus plain text:

  1. Counter (llm_requests_total): Monotonically increasing count of total API invocations partitioned by provider, model, and status.
  2. Histogram (llm_request_duration_seconds): Binned execution latency percentiles to compute p50, p95, and p99 in Grafana.
  3. Gauge (cache_hit_ratio): Dynamic float value representing the current semantic cache efficiency.
  4. Counter (llm_tokens_total): Total prompt vs completion tokens billed.

In Simple English: Kubernetes Health Probes (/healthz vs /readyz) & Prometheus

Never mix up these two probes in Kubernetes — mixing them causes cascading cluster outages:

  • /healthz (Liveness = "Is Python frozen/deadlocked?"): Should be an instant lightweight check returning 200 OK if the event loop is alive. NEVER check Redis/LLM API inside /healthz! Otherwise, when Redis blinks for 2 seconds, Kubernetes kills and restarts ALL your healthy AI pods simultaneously!
  • /readyz (Readiness = "Am I ready for user traffic right now?"): Checks DB & cache connections. If Vector DB isn't warm yet, Kubernetes safely holds traffic out of that pod until ready without killing it!