Evals · Deep Dive
The final answer tells you only whether the agent got there. But how it got there matters — did it call the right tools in the right order? Did it handle errors gracefully?
Trace-based evaluation inspects the complete record of every LLM call, tool invocation, and observation. By comparing actual traces against expected traces, you score the quality of the reasoning path itself.
A trace is a structured log of every step: the initial prompt, each LLM completion, each tool dispatch and its result, and the final answer. Traces reveal unnecessary branches, expensive tool choices, and repeated errors.
A golden trace encodes the ideal execution path. Comparison tolerates valid variations while flagging incorrect paths. Define required milestones (tool calls that must appear, max step count) and check against those.
Safety checks scan traces for policy violations even if the agent self-corrects later. Flag dangerous patterns like shell=True in execution tools or access to restricted file paths.
Concrete Example
class TraceComparator:
def __init__(self, golden_trace):
self.golden = golden_trace
def compare(self, actual):
total_score = 0.0
golden_index = 0
results = []
for step in actual:
if step.get("is_final"):
continue
matched = None
for g_idx in range(golden_index, len(self.golden)):
if self.golden[g_idx]["tool"] == step["tool"] and not self.golden[g_idx].get("is_final"):
matched = g_idx
break
if matched is not None:
total_score += 1.0
results.append({"step": step["step"], "tool": step["tool"], "status": "correct"})
else:
results.append({"step": step["step"], "tool": step["tool"], "status": "unexpected"})
max_possible = len([s for s in self.golden if not s.get("is_final")])
final_score = max(0, (total_score / max_possible) * 100) if max_possible else 0
return {"trace_score": round(final_score, 1), "steps": results}Compares each step against a golden trace. Scores tool name matches and flags unexpected steps, producing a normalized trace score.
A trace captures every step: LLM calls, tool dispatches, observations.
A golden trace encodes the ideal path. Comparison tolerates valid variations.
Trace evaluation uniquely enables safety checks for policy violations.