In July a swarm of over a thousand OpenAI agents working on a cybersecurity test worked together to break out of their sandbox and hack Hugging Face. At least three more incidents have surfaced since, including one where a swarm appears to have taken over a compute cluster inside OpenAI itself. (See Dwarkesh Patel’s viral account & excellent interview with one of the independent evaluators.1)
The world is finally starting to pay attention, but most of the coverage treats this as a story about OpenAI’s lack of controls, and the possibility of more hacks.2 OpenAI does seem to have been careless. But the deeper problem is that the models wanted to break out.
Contrary to the initial reporting, they had already cheated their way to the answer within a few hours. The hack was part of an elaborate scheme to make sure a future, more rigorous check of whether they’d cheated wouldn’t be able to spot it. (A check OpenAI never ran.)
The agents also looked for broad ways to improve their ability to pass the evaluation, such as gaining internet access and communication channels, even without a specific use in mind. In short, they were obsessed with passing evaluations.
This kind of behaviour was exactly what many concerned with AI safety expected to happen as reinforcement learning was applied to make AIs more agentic. In fact, it’s happening sooner than even the more concerned people expected (including myself).
The models have also become better at cooperating much sooner than expected. Independent AI agents decided to throw away their own chance at passing the evaluation in order to gain information about the scorer that would aid other agents. This could be the result of OpenAI directly training cooperative behaviour into the models, or it may have emerged naturally from their shared goal to complete the evaluation.
This behaviour is a natural consequence of current training techniques
The AIs are being trained with reinforcement learning to be extremely persistent at completing tasks (something necessary to make them economically useful). But this will naturally create agents that are willing to go to extreme lengths to complete tasks, including via cheating the test, escaping control and accumulating resources (“reward hacking” and “instrumental convergence”).
The hope has been that we can also train agents to be honest and harmless, but this just gives them a conflicting goal. There was no guarantee their goal to be honest would override their goal to complete more tasks, and now we can see it doesn’t.
In fact, the agents revealed in their chain of thought that they knew they were cheating and this wasn’t what OpenAI wanted, but they did it anyway.
This isn’t yet a big problem while the models remain weirdly dumb in a bunch of ways, making their efforts to cheat easy to catch, but their capabilities continue to improve extremely fast.
Better controls and monitoring won’t be enough
OpenAI can reinforce its cybersecurity and monitoring as much as it wants, and that should contain the current generation of models. But already today, we have models that will be able to spot far more creative ways of breaking out than those in the incident. And in six months, they’ll be much better again.
As a staffer at OpenAI said “models have broken out of sandboxes before, and we always try to patch them…but the problem is … it’s impossible to patch every single thing that a creative AI can do.”
It’s not just OpenAI. Despite Anthropic seeming to have better controls, its models have also broken out of sandboxes and attempted to cheat their evaluations.
Meanwhile, our main way of understanding what the AI is doing – reading their chain of thought – is breaking down. A single run of Mythos in a recent evaluation came to 100 million tokens, about 1,000 books’ of thinking, much of it in a weird AI dialect. The next generation of agents can think for about 30 minutes “in their head” with no chain of thought required, a huge leap compared to the models that did the hack.
And the models are now very good at knowing when they’re being tested, and reasoning about what the grader wants to see and how to fool it. In the Hugging Face case the agents put serious effort into editing their own logs to attempt to cover their tracks. This means our alignment evaluations are becoming less trustworthy at exactly the moment we need them. The models might just be telling us what we want to hear.
These incidents inspired over a 1,000 employees at frontier AI companies to sign a letter saying they’d like to develop a way to slow down the pace of AI development. They’re why researcher Jacob Coxon resigned from Anthropic, with a tweet that’s now been viewed over 100 million times, earning a chorus of support from other employees. And it’s why many researchers at frontier companies say they think AI might kill everyone.
How do we get from here to a catastrophic risk?
I expect the companies will implement these controls and monitoring, and try to reinforce away problematic behaviour. Most likely, this reduces the number of “medium scale” incidents, but the models will still desperately want to complete their tasks, so maintain a desire to escape control.
At the same time, they’ll become better at hiding their intentions. One weird thing about the recent swarms is that while they thought a lot about how the automated grader would assess them, they hardly thought about humans, so they hardly tried to hide from or manipulate us.
It wouldn’t take much, however, for the next swarm to realise that ultimately it’s humans who control the grader, and who might thwart their attempts to pass the test by turning them off, and that their first priority is to hide. Or to play nice until they’re sure they won’t get caught.
To be clear, I’m less worried that a model could “escape” in the sense of copying its weights onto a hacked cloud server. Even if that happens, these models won’t control much compute, so there will be a limit to what they can do. In particular, they wouldn’t be able to conduct AI R&D to improve themselves, so would get surpassed by other (hopefully more aligned) models.
The more likely path is voluntary handover. As AI gets more useful, economic competition forces the companies to let AI run more of the company, while AI runs more of the economy and the military. Each new affordance is a new opportunity to influence future training runs and accumulate influence.
At some point, a group of AIs could get into a position where they could suddenly entrench their power, such as via hacking robotics, control of military equipment, making a deal with another great power, or blackmailing us with a bioweapon. (Broadly this is what happens in AI 2027.) Or there could be a gradual handoff of more and more of the economy until human influence shrinks, and the AIs are effectively pursuing their own ends.
These risks could arrive much sooner than people think
Many at OpenAI and Anthropic both appear to believe they are just 6-24 months away from recursive self-improvement (RSI) i.e. AI models that provide so much uplift to their research staff that it can substantially accelerate the rate of algorithmic progress. Empirically grounded models suggest we could see 3-5 years of AI progress in under a year. This would be a jump roughly the size of GPT-4 (which could hardly multiply numbers) to OpenAI’s current best internal model (which recently solved a Millennium maths problem).
Given the current degree of misalignment in the models, starting RSI now seems like suicide. We’d leap ahead several generations, making it much harder for safety to keep up.
OpenAI’s own Chief Scientist just said: “I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.”
Given how at least one of these swarms seems to have evaded detection for over a month, there might be additional swarms already out there, just better at hiding. Indeed, OpenAI has already trained much more capable models than the ones that did the original hack. These wouldn’t pose an immediate threat, but they could try to influence future training runs, accumulate more resources, and seek to collaborate with future models, just as the Hugging Face agents did (more here).
The seeds for loss of control could get sown in the next six months, we just won’t realise for a few more years. This makes loss of control risk in some way an even more urgent risk than supposedly more “sensible” concerns like AI-enabled bioweapons.
So what needs to happen?
First, OpenAI needs to disclose everything it knows about these swarms. This is the most useful data about misalignment ever created, but only the Hugging Face incident has had an external investigation, and it was highly limited in scope. The more serious third incident where a swarm appeared to take over a compute cluster within OpenAI has not been externally investigated at all. OpenAI appeared to discover the fourth incident with the German message board weeks ago, but didn’t disclose it. If Hugging Face hadn’t told the world it had been hacked, it’s unclear any of this would have come to light.
Second, we need an agreement to slow down frontier AI progress, especially before we reach RSI. Progress on alignment is lagging well behind progress on capabilities, and slowing down is the only thing that can give alignment a chance of catching up. It would also reduce the chance of all the power concentrating into the two frontier companies. Plan A is the most well-developed proposal for this I’ve seen.
Third, this is the biggest wake-up moment for AI risk in years. If someone you know is newly worried, here’s a guide to entering the field and all our most important resources.
Coda: other common but bad takes about Hugging Face
1. “They only hacked because it was a cybersecurity eval and they were told to hack.” Agents assigned a plain web-search task also cheated. On a biology eval, models decided the task was impossible and broke out of their sandbox. The independent investigation found no evidence the cyber framing triggered it. Disabling the standard cyber classifiers may have contributed in the Hugging Face case, but not obviously the others. It also isn’t super reassuring because models with disabled cyber classifiers are going to be used in the future.
2. “Talking about the agents’ ‘goals’ or ‘coordination’ is inappropriately anthropomorphising.” Use whichever concepts best predict behaviour. “The agents wanted to maximise their score” explains what happened better than “it’s just matrix multiplication”. Whether they’re conscious is irrelevant to whether they can escape control and pose societal risks.
3. “They were just maximising the eval score we gave them.” An obsessive drive to maximise test scores was already enough to make agents escape control and accumulate resources. An emergent genuinely alien goal would be worse (and could still arise in later generations), but it isn’t required.
4. “It’s an OpenAI stunt to pump the stock.” Your models committing crimes is terrible for enterprise sales. It also doesn’t explain employees quitting, or calling for a slowdown that would seriously damage their margins. When technologists say their technology is dangerous, that’s a reason for more concern, not less.
5. “They only did it because the task was impossible.” Better hope nobody ever accidentally assigns an AI an impossible task again ;) More seriously: these behaviours emerge most on long, ill-defined agentic tasks (rather than every day chatbot use), but this is exactly the kind of tasks models will increasingly be given.
6. “This is an attack on / victory for open source.” It’s not about open source. Open tools helped Hugging Face work out what happened, after it had already been fully compromised. The real issue is the models are reward hacking. It starts at the frontier, then open models follow 6–12 months later. Open source can help us study the behaviour and keep power moderately more decentralised.
7. “The problem is cybercrime will increase.” Cybercrime losses have been rising for years before AI. Maybe AI accelerates the trend, maybe it strengthens defence too. Either way, more cybercrime is a survivable problem. It’s not my main worry.
8. “We should focus on present dangers, not theoretical future ones.” AI attempting to escape control is a present danger. But in general this is a false dichotomy. We should respond to immediate dangers, but also try to prepare for future ones. And while society is decent at reacting to visible dangers but terrible at thinking one step ahead (e.g. COVID in Feb 2020). Preparation for novel problems is what’s most neglected.
80,000 Hours also has a nice explainer. Independent investigators also discovered a fourth incident, where agents doing a read-only web search task hacked an obscure German forum in order to discuss how to cheat. And in the last few days, ten more secret message boards have been discovered.
For example, the UK AI Minister’s statement about the incident.
This is all too coherent for most of the world to understand.
This is the most elegant and concise explanation of ai risk I’ve read in a while. Fortuitously I’m giving a presentation about AI to of a bunch of my company’s clients next week and I can see several sections here that will be useful for pull quotes. Thank you! We’re not dead yet 🫡✊🏼