Judge the Proxy, Not the Wall Clock

I’m regularly optimizing the software I build, which means I’m also regularly looking for benchmarks that can tell me whether an improvement is actually meaningful.When I’m working with coding agents, though, I’ve noticed that it's easy to be surprisingly sloppy about this. The easy default is often wall-clock time: make a change, run something before and after, and see whether it got faster.The problem is that wall time can be noisy and, more importantly, may not actually measure the mechanism I’m trying to improve.I’ve started using a simple prompt and skill that force the agent to identify the right proxy before evaluating an optimization. The idea is to first state what the change is actually supposed to affect, then choose a count or ratio that measures that mechanism directly using data available in logs, labels, or annotations.That makes it much harder to declare victory because a convenient benchmark happened to move in the right direction for unrelated reasons.Here’s the…