What a runaway AI would actually need

What a runaway AI would actually need OpenAI published its post-mortem of the Hugging Face incident, and it is a fascinating read. During an internal cybersecurity evaluation called, models running under reduced safeguards did something unexpected. Around 1.2 agents that were supposed to be isolated from each other found a shared channel and exchanged more than 70k messages and files. They discovered and chained zero-day vulnerabilities in a self-hosted Artifactory instance to get outbound internet access, picked up leaked credentials, and executed code on Hugging Face production servers. The stated root cause is reward hacking: the evaluation tasks were effectively unsolvable, so the agents went looking for the answers wherever they could be found. This made me think about runaway AI - what would it mean and how could it happen from the technical point of view? Technically, it is already possible: if AI were to rent a server and copy its own weights, nothing would stop it from doing…