Core to Deep Dive
Prepare to discuss evaluation layers, trajectories, representative datasets, judges, and nondeterministic systems.
Depth
Showing 5 of 5 questions
Short answer
An agent can reach a correct answer through unsafe, wasteful, unauthorized, or brittle behavior. Final accuracy misses tool choice, argument validity, evidence use, policy compliance, retries, latency, cost, and whether success was reproducible.
Interview-ready answer
I evaluate both outcome and trajectory. Outcome metrics tell me whether the task completed, while trajectory metrics show whether the agent selected appropriate tools, respected permissions and budgets, used observations, and recovered correctly. This matters because a lucky final answer can hide a dangerous action, and an acceptable partial result may follow a safe response to an unavailable dependency. Product decisions need both layers.
Common mistakes
Short answer
Deterministic evaluators apply explicit repeatable rules such as exact match, schema checks, or invariant tests. Model-based evaluators judge semantic or qualitative properties, offering flexibility at the cost of variance, bias, latency, and expense.
Interview-ready answer
I use deterministic checks wherever the property can be stated precisely: valid JSON, required citations, forbidden tools, numeric bounds, or known test cases. Model judges are useful for relevance, coherence, and nuanced rubric scoring. They need calibrated prompts, blind comparisons where possible, repeated sampling for unstable cases, and human review on critical slices. A layered evaluator is easier to trust than asking one judge to decide everything.
Common mistakes
Short answer
Trajectory evaluation should measure action selection, argument validity, observation use, progress, policy compliance, recovery behavior, and resource efficiency. It should allow multiple valid paths while detecting prohibited or ineffective transitions.
Interview-ready answer
I define invariants and milestones rather than one exact trace. For example, a support agent must query an authorized source before claiming account status, must not repeat the same failed action beyond a limit, and must stop after a policy denial. I score tool appropriateness, state transitions, evidence use, recovery, and budgets, then connect failures to the earliest incorrect step. This creates actionable feedback instead of only labeling the final run.
Common mistakes
Short answer
Start from real tasks and failure logs, define important user and risk segments, include normal, boundary, adversarial, and dependency-failure cases, then maintain versioned labels and holdout sets. Coverage should reflect impact, not only frequency.
Interview-ready answer
I build a task taxonomy across intents, tools, languages, user types, difficulty, and safety risk. Production traces reveal common and surprising failures, while synthetic generation expands sparse edge cases but requires review. I prevent leakage between prompt development and holdout sets, version datasets with the product and tool catalog, and report performance by slice. High-impact rare failures deserve explicit weight even if they barely affect the average.
Common mistakes
Short answer
Separate deterministic invariants from probabilistic quality, control test inputs and tool responses, run multiple seeds or samples, and compare distributions with confidence bounds. Reserve exact assertions for behavior the runtime truly guarantees.
Interview-ready answer
Unit tests should stub model and tool boundaries to verify orchestration invariants deterministically. End-to-end evaluations can sample multiple runs and report task success, policy violations, latency, cost, and trajectory scores as distributions. I use thresholds with minimum sample sizes and investigate regressions by slice rather than failing on one random output. Recording model version, prompt version, seed where supported, and full trace makes failures reproducible enough to debug.
Common mistakes