Agents Should Earn Authority, Not Inherit It

Much of my career has been spent deciding what software is allowed to do, and every system I worked on eventually had to answer the same question, which is what happens when the code in front of you does something you did not expect. Agents have made that question urgent again, and most of the conversation about them is about making the model behave. I think that is the wrong place to start. As I put it in Containing the Optimizer, safety is a design problem, not a policy problem. The more useful question is what the model can reach when it does not behave, and the easiest way to show why is to walk through a run. An agent should be free to decide how to accomplish a task. It should not be free to decide what it is authorized to do. A flaky test is enough Here is what a run looks like in a system designed this way. Take an engineer on a payments team with a refund test that fails about one run in five. The test submits the same refund twice with one idempotency key and asserts that…