Human-AI partnerships are for alignment, not capability

It’s common to compare the current AI takeover of software engineering to the rise of AI in chess. Chess AIs went from much weaker than serious players to much stronger than even the strongest humans. Between those points, there was a middle period dominated by “centaurs”: human-AI partnerships that were stronger than unassisted AIs or humans. Lots of people think that we’re currently in a world of software engineering centaurs. According to them, coding AIs are not yet capable enough to replace engineers, but AI-assisted engineers are better at programming than both AIs and humans. This is partialy correct, but the wrong way to think about it. AI-assisted engineers are better, but unlike with chess centaurs, they’re not actually better at programming. When I ask agents to write code, they make fewer mistakes than I do1 and are orders of magnitude faster. The code that they write always compiles, rarely has race conditions or other concurrency errors, works on mobile browsers, and so…