Core to Deep Dive
Review traces, logs, metrics, cost, latency, replay, and debugging methods for multi-step agent workflows.
Depth
Showing 5 of 5 questions
Short answer
Agent observability is the ability to understand a run through structured traces, logs, metrics, inputs, outputs, tool calls, model usage, and state transitions. It supports debugging, evaluation, operations, and cost control.
Interview-ready answer
Unlike a single API request, an agent run can fail across planning, retrieval, tools, state, or policy. I model the run as a trace with spans for model calls, tool calls, retrieval, validation, and approvals, all tied to versions and a correlation ID. Metrics aggregate latency, cost, success, and failure classes, while secure logs retain enough context to reproduce issues without exposing secrets or sensitive user data.
Common mistakes
Short answer
Log a run ID, actor and tenant context, versions, timestamps, model and tool spans, sanitized inputs and outputs, token and cost usage, state transitions, retries, approvals, errors, and final status. Apply redaction and retention policies.
Interview-ready answer
I want enough data to reconstruct control flow without turning logs into a sensitive-data dump. Each event includes correlation IDs, parent span, component version, duration, status, and typed attributes. Prompts, retrieved context, and tool payloads are redacted or referenced through controlled storage. I also log policy decisions and user approvals because they explain why an action was allowed or denied.
Common mistakes
Short answer
Traces preserve ordered spans, inputs, outputs, state changes, timing, and errors, so you can find the earliest incorrect transition rather than blaming the final response. Parent-child relationships show downstream effects.
Interview-ready answer
I compare expected invariants with the trace and locate the first divergence: wrong retrieval, malformed tool arguments, stale state, ignored observation, or policy rejection. Span timing separates slow model calls from tool timeouts, while version metadata shows whether a deployment changed behavior. A good trace viewer also links evaluation feedback to exact events, turning a vague bad answer into a concrete repair target.
Common mistakes
Short answer
Track end-to-end and per-step latency, token and tool cost, task success, groundedness or rubric scores, policy violations, tool error rates, retry counts, timeout rates, abandonment, and availability. Segment metrics by task and risk.
Interview-ready answer
I use percentiles rather than averages for latency and cost, and decompose them by model, retrieval, and tool spans. Quality metrics depend on the task but should include both outcome and trajectory. Reliability includes dependency failures, retries, loops, fallbacks, and terminal statuses. Business or user metrics such as completion and escalation matter too, because a technically valid response may still be unhelpful.
Common mistakes
Short answer
Capture comparable successful and failed traces, control model and tool variability, segment by versions and dependencies, and locate the earliest divergent state. Reproduce with recorded inputs and deterministic tool fixtures before changing prompts.
Interview-ready answer
I first define the failure precisely and query traces by task, model version, tool version, latency, and error class. I diff a failed run against a nearby success to find whether retrieval, planning, tool output, context size, or timing diverged. Then I replay with frozen tool responses and a controlled model stub to isolate orchestration, followed by sampled live-model runs. The fix gets a regression case and a production monitor for recurrence.
Common mistakes