The Secret To Long-Horizon Agent Autonomy? Silent Failing

A large and growing body of research and industry data points to an inevitable conclusion: reliable long-horizon autonomous coding agent operation is highly improbable. And yet, pretty much on a daily basis, I see people claiming online to have achieved it. They let their agents run for hours or even days, generating complex software without – if they’re to be believed – any human intervention. No doubt some of them are just plain lying to impress their friends (or their boss, or a potential employer or customer looking for someone who’s cracked The Secret Code). But I suspect at least some of them genuinely believe it. And, given how improbable that is, it’s worth looking a little deeper at these claims. In the instances where I’ve been reasonably assured that they did indeed let the agent(s) run without any intervention – and the Clever Hans effect is difficult to rule out – it’s turned out there’s a very simple explanation. The agent(s) didn’t do the work reliably. Mutation testing…