Track
Trace · Log · Replay · Improve
A working agent demo is not enough. In production, agents fail unpredictably — they take wrong actions, miss tool arguments, hit edge cases, or behave differently than expected. Without observability, you are debugging blind.
Observability is how you understand why an agent behaves the way it does. By logging every step, tracing every tool call, and measuring latency and cost, you build a complete record of each run. Interviewers ask about observability to see if you've actually deployed agents.
Every agent run produces a sequence of steps. Per step, log: timestamp, step number, raw LLM input/output, tool name and arguments, tool duration in ms, tool result or error. Use JSON Lines format with run_id and step_id for correlation.
A tool trace is the sequence of tool calls over a run — it reveals behavioral patterns. Replayable runs store full input/output per step so you can replay deterministically against a new version for regression testing.
Track tokens per call, number of calls, tool execution costs, and total run duration. Set latency budgets and alert when exceeded. Debugging workflow: identify failed runs via metrics → open trace → find root cause → fix → verify with replay.
Concrete Example
import time
class TraceLogger:
def __init__(self, run_id):
self.run_id = run_id
self.steps = []
def log_step(self, step_num, llm_output, tool_name, tool_args, tool_result, duration_ms):
entry = {
"run_id": self.run_id,
"step": step_num,
"timestamp": time.time(),
"llm_output": llm_output,
"tool_call": {"name": tool_name, "args": tool_args},
"tool_result": tool_result,
"duration_ms": duration_ms,
}
self.steps.append(entry)
return entry
def summarize(self):
tool_calls = [s for s in self.steps if s["tool_call"]["name"]]
return {
"run_id": self.run_id,
"total_steps": len(self.steps),
"tool_calls": len(tool_calls),
"total_duration_ms": sum(s["duration_ms"] for s in self.steps),
"last_step": self.steps[-1] if self.steps else None,
}Captures every step with timestamps and durations. Each step records LLM output, tool call, and tool result. summarize() aggregates into a high-level report for debugging and cost analysis.
Log every step with run_id, timestamps, LLM calls, and tool results in JSON lines.
Capture the full sequence of tool calls for behavior analysis.
Store enough context to replay runs deterministically for regression testing.
Track token usage and tool costs per run.
Measure step durations to identify slow tools or bottlenecks.
0 problems. Sign in to start solving.
Sign in to open a workspace and solve these problems.