Track
Measure · Judge · Trace · Improve
How do you know if your agent is actually working? A single demo run looks promising, but what about edge cases? What about the time the agent silently ignored its instructions? What about prompt injection?
Evaluation is how you get answers to these questions. Not by reading outputs and guessing, but by running systematic checks that measure specific behaviors — correctness, tool usage, safety, trajectory quality.
The simplest eval: compare the agent's output to a known reference. But exact string comparison is too brittle — 'Hello!' and 'hello' should match. Normalization lowers case, strips punctuation, and collapses whitespace before comparison.
Both the prediction and the reference go through the same normalization pipeline. This catches format differences that would break naive string comparison.
Exact match only checks the final answer. For agent behavior, score multiple dimensions separately: correctness, tool usage, error handling, and efficiency. Each dimension gets a score, and the overall score is a weighted combination.
Final answer evals compare output to expected answer — fast to write but blind to behavior. Trace evals examine the full trajectory: which tools were called, in what order, with what arguments.
Use both: final answer evals for regression coverage, trace evals for safety and behavioral alignment. In interviews, be ready to design both layers.
Binary pass/fail cannot tell you which dimension needs improvement. Partial credit breaks evaluation into correctness, tool usage, safety, context handling, and efficiency. Each dimension gets a score and weight, producing a dimensional profile.
Hidden tests prevent overfitting prompts to specific cases. They cover edge cases, prompt injection, invalid tool calls, and permission violations. Report results safely: 'Failed 2 of 5 hidden safety tests' without revealing exact inputs or outputs.
Concrete Example
import string
_PUNCT = str.maketrans("", "", string.punctuation)
def normalize(s):
return " ".join(
s.lower().translate(_PUNCT).split()
)
def exact_match(preds, refs):
if not preds:
return 0.0
matches = sum(
1 for p, r in zip(preds, refs)
if normalize(p) == normalize(r)
)
return matches / len(preds)Both predictions and references pass through the same normalizer: lowercase, strip punctuation, collapse whitespace. Returns accuracy in [0, 1].
Fast, deterministic comparison after normalization.
Multi-dimensional evaluation that profiles agent strengths.
Final evals check output; trace evals check behavior. Both are needed.
Weighted scoring across dimensions reveals progress pass/fail hides.
Held-out cases that measure generalization and prevent overfitting.
Combine exact match, rubric scoring, trace evals, and hidden tests.
Explore these deeper articles to master specific topics in Evals.
Interview revision
8 problems. Sign in to start solving.
Sign in to open a workspace and solve these problems.