When the unit of work is a multi-step plan, the trace becomes the product. Patterns we use for evals, replays, and the post-mortems that follow.
Traditional observability assumes a request is short, mostly deterministic, and either succeeds or fails. Agentic workloads break all three assumptions. A single “request” might be a thirty-step plan that calls six tools, changes its mind twice, and ends in a state nobody anticipated. Your dashboards weren't built for that.
The shift that helped us most: stop thinking of a run as a log stream and start thinking of it as a tree you can replay. Once the trace is a first-class artifact, evals, debugging, and post-mortems all become the same activity viewed from different angles.
01 — The trace is the productMake runs replayable
Every step records its inputs, its outputs, the model and parameters, and a content hash of the prompt. Given those, any run can be re-executed deterministically against a different model or a patched tool. This single property — replayability — is what turns “it failed in prod” into a unit test.
// every step is content-addressed and replayable
type Step = {
id: string,
parent: string | null,
promptHash: string,
tool: string | null,
input: Json,
output: Json,
model: "claude-...",
}If you can't replay a run, you can't debug it — you can only guess at it.
02 — Bisecting failuresFrom regression to root cause
Because runs are deterministic given the trace, a regression becomes a bisect. Pin the inputs, sweep the variable — model version, temperature, a single tool's behaviour — and binary-search to the step that changed. We've turned multi-day “why is the agent worse now” investigations into an afternoon.
- Pin the seed. Same inputs, same tools, one variable at a time.
- Diff the tree, not the text. Structural diffs surface the step that diverged.
- Keep the failures. Every regression becomes a permanent eval case.
The payoff isn't a prettier dashboard. It's that the team stops being afraid to change the prompt.