AI agent evals for interviews and reliable systems

Begin with a concrete success policy. State what the agent should accomplish, which tools it may use, what evidence it must cite, and which actions are prohibited. Metrics are useful only when they represent this policy.

Use category-level scores when tradeoffs matter. A task can be correct but unsafe, grounded but unnecessarily expensive, or efficient but incomplete. Preserve these distinctions instead of collapsing them immediately into one number.

Output checks include exact match, structured-field validation, semantic criteria, citation verification, and rubric-based judgments. Trajectory checks inspect tool selection, argument validity, ordering, retries, observations, and terminal states.

Run important cases more than once and report pass rates or score distributions. Fix the model version, prompt version, tool definitions, and sampling settings in the evaluation record so changes are attributable.

Averages can hide rare severe failures. Track worst-case categories and safety violations separately, and define release thresholds appropriate to the consequence of error.

Start with a small reviewed set covering normal, edge, and adversarial cases. Run it in development and before releases. Sample production failures carefully, remove sensitive data, label them, and promote representative cases into the regression suite.

During interviews, be explicit about judge limitations, data leakage, annotation disagreement, and monitoring after deployment. Evaluation is an operating process, not a one-time benchmark.