Week 18 • Day 2Week 18

Day 2:Production LLM Observability — Langfuse Trace Schemas & Feedback

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 2: Production LLM Observability — Langfuse Trace Schemas & Feedback

1. First-Principles Problem: Why Traditional APM Fails for LLMs

Traditional Application Performance Monitoring (APM) tools (Datadog, New Relic, Dynatrace) excel at measuring:

  • HTTP 200 OK vs HTTP 500 Error
  • CPU & RAM usage
  • Total request duration in milliseconds

The AI Blind Spot

In an AI application, an endpoint can return HTTP 200 OK in 1,200ms, yet be a complete catastrophic failure:

  1. The model hallucinated a non-existent AWS command.
  2. The model violated safety guardrails.
  3. The prompt consumed 15,000 tokens when 500 were needed.
  4. The user gave a thumbs-down score.

Traditional APM marks this transaction as 100% green and healthy!


2. The Langfuse Trace Data Model

Dedicated AI observability platforms (Langfuse, Arize Phoenix) structure telemetry around AI-native entities:

Architecture Flow Diagram
Trace {
+String id
+String userId
+String sessionId
+List~String~ tags
+Dict metadata
}
Span {
+String id
+String name
+Float durationMs
+Dict input
+Dict output
}

Key Native Entities:

  1. Trace: Represents the end-to-end user transaction (user_id, session_id, tags).
  2. Span: Measures non-LLM operations (e.g., vector DB retrieval, reranking, tool execution).
  3. Generation: Records the exact LLM prompt, completion, model version, token breakdown, and exact dollar cost.
  4. Score: Quantifies human or automated evaluations (user_feedback: +1 or -1, relevance: 0.92, toxicity: 0.00).

In Simple English: Tracing AI Thinking Steps (Langfuse)

Traditional web monitoring (Datadog APM) only answers "Did /api/chat return HTTP 200 in 800ms?" It has NO idea whether the answer was smart or a total hallucination! LLM Observability (Langfuse) records a nested tree for every user request:

  • Trace: Whole user session (cost=$0.004, user_thumbs_up=1)
    • Span 1 (Retrieval): Found 3 chunks in 35ms
    • Generation (LLM Call): Exact prompt text, output tokens, temperature, USD spend, and user feedback score linked straight back to prompt version v4!