The real AI risk is inside the labs

Amodei in his latest blog post wrote a mix of agreeable things and things that I believe misrepresent where the real risk of AI is located. I want to focus my attention on why, among all the risks, open weight models constitute the mildest one. I write these words as a person who strongly believes AI may be very dangerous in the near future: 1. Exactly like what happened during the OpenAI / HF incident (which was a joke, but focus on the modalities, not the outcomes), the first serious AI incident is very likely to happen *inside* the walls of frontier AI labs, while testing a new model, or while the AI lab employees, or the few externals who have access, do something wrong compared to the expected power of the model. 2. Closed models that will never even be opened to the public will be just a few TBs of data. All you need to leak one is a single person with access and the wrong goals, and you are back in the situation of open models. Open models are released *after* testing, and…