Day 6: Evaluating Agents — Trajectory Metrics, Tool Selection F1 & Step Efficiency

1. First-Principles Agent Evaluation: Why Final-Answer Grading Is Not Enough

An agent can arrive at the right final text answer by luck (e.g., guessing that a service is healthy after calling the wrong tool or passing the wrong namespace) or via an inefficient 12-step detour when 2 steps were optimal.

Therefore, production agent evaluation grades the Execution Trajectory $\tau = [(t_1, a_1), (t_2, a_2), \dots, (t_m, a_m)]$ against a reference Golden Trajectory $\tau^* = [(t^*_1, a^*_1), \dots, (t^*_k, a^*_k)]$:

1. Tool Selection Precision, Recall & $F_1$: $\text{Precision} = \frac{|T_{\text{pred}} \cap T_{\text{gold}}|}{|T_{\text{pred}}|}, \quad \text{Recall} = \frac{|T_{\text{pred}} \cap T_{\text{gold}}|}{|T_{\text{gold}}|}, \quad F_1 = \frac{2PR}{P + R}$
2. Argument Exact Match Rate: Fraction of expected (tool, args) calls where all required arguments match the golden specification.
3. Trajectory Optimality Ratio (Step Efficiency): $\eta_{\text{steps}} = \min\left(1.0, \frac{|\tau^*|}{|\tau|}\right)$ If $\tau^*$ requires 2 steps and the agent takes 5 steps, $\eta_{\text{steps}} = 2 / 5 = 0.40$.

In Simple English: Grading the Whole Path, Not Just the Final Answer (Trajectory Metrics)

Suppose two agents both correctly tell a user "Payments service is fixed":

• Agent A called [check_pod_logs, restart_worker] (2 steps, 100% optimal!).
• Agent B panicked, called [list_all_buckets, delete_temp, check_pod_logs, grep_wrong_file, restart_worker] (5 steps, 3 useless tools!). Trajectory Evaluation grades HOW the agent arrived:
• Tool Precision & Recall ($F_1$): Did it pick the right tools without noisy wrong calls?
• Step Efficiency ($\eta_{\text{steps}} = \min(1.0, |\tau^*| / |\tau|)$): Agent A took $2/2 = \mathbf{100%} efficient; Agent B took \2/5 = \mathbf{40%}$ efficient!