Out-Of-Distribution Errors – What Might Be Lurking In The Long Tail?

In my previous post, I derived a formula from my Special Theory of Autonomous Agent Reliability that predicts the reliable horizon over which a coding agent can operate before Actual Intelligence needs to step in to check for unanticipated errors. I theorised that the reliability of a single step – a model interaction – in an agentic workflow is: R = 1 – (1 – C)(1 – P) Where C is the probability of the model getting it right, and P is the probability of errors being caught before they propagate and compound. 1 – P is doing a lot of the work here. It represents the probability of errors our automated quality checks didn’t anticipate – errors that are outside their distribution (we didn’t write a test for that, the linter doesn’t check for that, the model wasn’t trained on that, etc). Those of us who care about reliability – and I appreciate we are rare beasts, verging on extinct, in 2026 – might well ask “What’s in that long tail of out-of-distribution errors?” These are the errors…