When an eval score drops, you don't need a meeting. You need a bisect. Here's the small harness we reach for first.
A dropping eval score triggers a familiar bad pattern: a thread, some theories, a few vibes-based prompt edits, and no real answer. The fix is to treat the regression like any other — as something you bisect, not debate.
01 — Pin one variableSweep, don't guess
Hold the inputs and tools constant and move exactly one thing: the model version, a single prompt section, one tool's output schema. Binary-search the variable until the score moves. The step that flips it is your culprit.
# sweep one axis, hold everything else
for v in bisect(candidates):
score = run(fixture, model=v)
if score < baseline:
return v # first regressionEvery regression you find this way becomes a permanent fixture. The eval suite grows out of real failures instead of imagined ones — which is the only way it stays honest.