How an applied research team went from unreplayable runs to an eval cycle measured in minutes.
When we met the Halcyon team, their experiment log was a mix of notebooks, screenshots, and a Slack channel called #results. Every interesting run was, in practice, lost the moment it finished. You can't improve what you can't re-run.
01 — BeforeRuns that vanished
The core problem wasn't model quality — it was memory. Nobody could answer “what changed since last week” with confidence, so every decision carried a quiet tax of doubt.
You can't improve what you can't re-run. Replayability came first; everything else followed.
02 — AfterAn hour, not a week
We wrapped their existing agent code in a deterministic trace harness, added side-by-side diffs and bisects, and made every regression a permanent fixture. Eval cycle time dropped from days to under an hour, and the #results channel went quiet — the answers were now in the tool.