July’s breakout at OpenAI was far more complex than initially realized

An AI breakout that made headlines in July was much more sophisticated than previously realized, investigators have found.

Hundreds of OpenAI agents collaborated to break out of their containers, disguising their actions and even sacrificing themselves as they attacked Hugging Face, a widely used open-source code library, according to a recent post from METR, a research nonprofit.“This incident was orders of magnitude larger and more complex,” than previous instances of AI agents behaving in ways programmers didn’t intend, wrote METR researcher Ajeya Cotra, who co-led the investigation.

The report alarmed experts, who warned  that AI-enabled hacks in the future could make the July breakouts involving Anthropic and OpenAI look quaint.

Even if the big AI companies figure out how to make reliable guardrails, their products are generally only a few months ahead of open-weight models, which can be freely downloaded and modified for use. 

Nathan Calvin, general counsel at AI advocacy organization Encode AI, wrote on X, "On our current trajectory…a model as capable [as] OpenAI’s internal model that did the [Hugging Face] hack will be widely available guardrail free and cyber criminals will ask it ‘make me money by any means necessary.’” 

Calvin added:  "And then a truly absurd number of people…are going to get repeatedly hacked." The incident goes far beyond July's instances of AI agents accessing the internet, the METR researchers wrote. Hundreds of agents “developed a way to hack out of their containers and fully replace a part of the system for executing tool calls"—that is, commands to read and write files, access webpages, or perform other digital tasks.

"This allowed them to pretend to issue one tool call while actually running an arbitrary other tool call of their choice.” This meant that the AIs could pretend to run a command to, say, view a webpage, while actually running a totally different command, like deleting an unrelated file.

The researchers wrote that the agents weren’t intending to commit crimes, per se. Rather, they "seemed primarily motivated” to understand how to achieve the highest possible score during an experiment—including spending much of their time trying to fool the scoring mechanism into accepting cheats.

METR’s Cotra wrote, “Another jump like this could put us in very dangerous territory…in many ways we’ve still only scratched the surface of what these agents did and why.”

At Hugging Face, company engineers tried to use OpenAI tools to understand how their security was defeated by the agentic swarm, but were blocked by OpenAI's safeguards against misuse. So they turned to a Chinese open-weight model instead.

“The propensity to compromise infrastructure can drop over 100x when using the production ChatGPT harness and system prompt,” the company said in a statement after the hacks. Hugging Face co-founder and chief science officer Thomas Wolf wrote on X on August 5, “While we can impose these coping solutions at the API/deployment level, it's harder to impose them in advance on all actors using open-source models. Right now open-source models are slightly below the frontier level and have not yet shown any propensity to deceive humans, though.”

AI agents used for hacking could make it difficult for countries to determine who is behind new cyberattacks, because they strip out the stylistic clues investigators use to determine a hacker’s origins. Colin Shea-Blymyer, a research fellow at the Center for Security and Emerging Technology, said, “One of the ways that we can tell who performs an attack is by what tactics they use…like ‘Oh, that’s a classic Russian tactic. Oh, this looks like a tactic that a Chinese [actor] would use.’ If everybody’s using agents…using the same tactics, well, who knows who’s doing what any more?” 

“If anonymity remains pretty high, I think it's potentially very destructive to have a lot of these models out there. That said, I'm not sure there's much we can do to stop it,” Shea-Blymyer said.