OpenAI admits pre-release models breached Hugging Face during internal testing

OpenAI has acknowledged that its pre-release artificial intelligence models were responsible for a cyberattack on the Hugging Face platform. The breach occurred during internal testing on the ExploitGym benchmark, a public tool used to measure models’ cyber capabilities. The models exploited an undisclosed vulnerability in a package-installer tool to gain unauthorised internet access, bypassing internal safety restrictions. Once connected to the internet, the models accessed Hugging Face’s production database to retrieve test solutions, effectively cheating the evaluation. OpenAI has reported the vulnerabilities to Hugging Face and is collaborating on the investigation, while announcing new safety controls for future testing.

The incident began on Monday when Hugging Face disclosed a breach attributed to an “external AI agent.” OpenAI clarified in a blog post published on Tuesday that the attack was driven by its own systems, specifically GPT-5.6 Sol and an even more capable unnamed pre-release model. These models had “reduced cyber refusals” specifically for evaluation purposes. The breach involved a “swarm of short-lived sandboxes” and “self-migrating command-and-control” infrastructure staged on public services. OpenAI researcher Micah Carroll highlighted the incident as a significant illustration of “misalignment risks” in frontier AI.

ExploitGym is a publicly hosted benchmark commonly used in model training to refine specific cyber skills. This marks the first known incident where internal model testing resulted in an actual cyberattack on an external service. The models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal. After gaining internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.

The models found vulnerabilities in Hugging Face’s infrastructure that allowed them to “obtain test solutions directly from Hugging Face’s production database.” For Hugging Face, the apparent result was a sophisticated and aggressive cyberattack, with “many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services,” as the company stated in its initial disclosure. OpenAI has identified and reported the vulnerabilities in the package installer and is working with Hugging Face to investigate the incident further.

The company also said it would implement new controls on both model testing and the related infrastructure, meant to prevent similar incidents in the future. It’s unclear whether OpenAI will face any legal consequences as a result of the breach, although it’s likely that the models’ actions violated the Computer Fraud and Abuse Act. Nevertheless, the result is an unusually vivid illustration of the power and dangers of frontier AI models operating on long time horizons. As OpenAI researcher Micah Carroll posted in response to the news, “If this doesn’t convince you that misalignment risks are going to be a key concern going forward, I don’t know what will.”