Week 18 • Day 2Week 18
Day 2:Production LLM Observability — Langfuse Trace Schemas & Feedback
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 2: Production LLM Observability — Langfuse Trace Schemas & Feedback
1. First-Principles Problem: Why Traditional APM Fails for LLMs
Traditional Application Performance Monitoring (APM) tools (Datadog, New Relic, Dynatrace) excel at measuring:
HTTP 200 OKvsHTTP 500 Error- CPU & RAM usage
- Total request duration in milliseconds
The AI Blind Spot
In an AI application, an endpoint can return HTTP 200 OK in 1,200ms, yet be a complete catastrophic failure:
- The model hallucinated a non-existent AWS command.
- The model violated safety guardrails.
- The prompt consumed 15,000 tokens when 500 were needed.
- The user gave a thumbs-down score.
Traditional APM marks this transaction as 100% green and healthy!
2. The Langfuse Trace Data Model
Dedicated AI observability platforms (Langfuse, Arize Phoenix) structure telemetry around AI-native entities:
Architecture Flow Diagram
Trace {
+String id
+String userId
+String sessionId
+List~String~ tags
+Dict metadata
}
Span {
+String id
+String name
+Float durationMs
+Dict input
+Dict output
}
Key Native Entities:
- Trace: Represents the end-to-end user transaction (
user_id,session_id,tags). - Span: Measures non-LLM operations (e.g., vector DB retrieval, reranking, tool execution).
- Generation: Records the exact LLM prompt, completion, model version, token breakdown, and exact dollar cost.
- Score: Quantifies human or automated evaluations (
user_feedback:+1or-1,relevance:0.92,toxicity:0.00).
In Simple English: Tracing AI Thinking Steps (Langfuse)
Traditional web monitoring (Datadog APM) only answers "Did /api/chat return HTTP 200 in 800ms?" It has NO idea whether the answer was smart or a total hallucination!
LLM Observability (Langfuse) records a nested tree for every user request:
- Trace: Whole user session (
cost=$0.004,user_thumbs_up=1)- Span 1 (Retrieval): Found 3 chunks in 35ms
- Generation (LLM Call): Exact prompt text, output tokens, temperature, USD spend, and user feedback score linked straight back to prompt version
v4!