Week 16 • Day 3Week 16
Day 3:OpenTelemetry Agent Tracing, Hierarchical Spans & Audit Logs
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Day 3: OpenTelemetry Agent Tracing, Hierarchical Spans & Audit Logs
1. First-Principles Observability: Why Flat Logs Fail for Multi-Agent Systems
When a user request triggers a Supervisor Agent Semantic Tool Router Sub-Agent 3 parallel MCP Tool calls, flat print() or single-line logs make it impossible to answer:
- Which sub-agent caused 80% of the p99 latency?
- Which tool call failed and triggered a retry?
- Has the audit log been tampered with after a security incident?
Hierarchical Distributed Trace Tree (OpenTelemetry GenAI Conventions)
Every end-to-end execution shares a single trace_id, while each operation (agent.run, llm.chat, mcp.tool.call, guardrail.check) creates a Span:
span_id: Unique 16-hex identifierparent_span_id: Links child tool calls to the parent agent turnattributes:gen_ai.usage.input_tokens,gen_ai.usage.output_tokens,mcp.server,mcp.toolaudit_hash: SHA-256 hash chain providing tamper-evident compliance logging.
In Simple English: Flight Recorder Logs (OpenTelemetry Hierarchical Spans)
When a 5-step multi-agent workflow fails silently for 1 out of 1,000 users, plain print() statements are useless!
Distributed Tracing (OpenTelemetry) records a nested family tree (trace_id + parent_span_id):
- Root Trace (
Incident Resolution: 1,420ms, $0.008)- Span 1:
Retriever (BM25 + Dense): 42ms - Span 2:
Worker LLM Call: 980ms, 1,240 tokens - Span 3:
MCP Tool Execution: 310ms, status=OKEvery failure pinpoints the exact slow/broken step down to the millisecond!
- Span 1: