On Inevitability

There’s a widespread belief that artificial superintelligence (ASI) is inevitable. In this post, I argue that this is false, and believing it is upstream of a lot of bad discourse. Why do people believe it? It’s easy to believe. Samuel Butler, already in 1863, observed that machines are improving faster than humans are improving. Today, AI systems are gaining capabilities much faster than humans: in five years we went from systems that can barely string together a sentence to autonomous cybercrime. Biological nervous systems have many virtues, like being power-efficient (because they run on Brownian motion, i.e. for free), but they were designed by evolution, a greedy optimizer that can’t plan or readily backtrack. Given what we know about the physical limits of computation and exploratory engineering, it is reasonable to think we can do better, and build machines that do the work of the brain, but better, faster, cheaper, and with fewer transitive dependencies (a single human needs…