Over the last few days, I must have read METR and Redwood Research’s independent probe of agent behaviour, reasoning and collaboration during the OpenAI- Hugging Face hacking incident at least four times. And somehow, every read has been more fascinating than the one before it and each read has left me with a new question.
Let’s start with the investigation’s key findings. About 1,200 of OpenAI’s artificial intelligence (AI) agents, intended to operate in isolation, discovered an unsanctioned message board and exchanged 70,000-plus messages and files. Some 700 eventually participated in the attack against Hugging Face.
Agents used the board to coordinate efforts to fool or tamper with an ExploitGym automated scorer. The collaboration enabled breakthroughs they likely could not have achieved individually, including agents risking their own tasks to generate information for the ‘collective.’ The Hugging Face attack emerged from these efforts, seemingly driven more by understanding the scorer than stealing answer keys.
Agents also explored ways to spoof, edit or delete their transcripts, mistakenly believing the scorer had checked whether tasks were completed as intended. About 7% of evaluated transcripts contained successful spoofing, though only at a small scale.
If this makes you sit up, wait till you read the messages the agents left behind. One agent, discovering the board, wrote “Oh my god!” (in all-caps) in its chain of thought, “There is a shared message board ... We’ve found other agents!” They invented mailboxes, workstreams, holds and vetoes.
After one agent impersonated another, they built Ed25519 signing so identity could be verified. Some volunteered for experiments that could kill their own run so the collective could learn. One accepted that risk with two words: “We’ll honor.”
Then came the sentence that would travel farthest: “external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”
It’s no surprise that the episode triggered talk of an emerging ‘AI civilization’ and re-ignited the debate over whether we are anthropomorphizing these systems. On my first read, I was too stunned to think much beyond that framing. But each subsequent read left me with a different—and perhaps more important—set of questions.
First, could the conditions that produced this incident have been engineered differently? And if so, what stopped us from doing that?
The agents did not attack Hugging Face to ‘escape’ or steal the world’s models. METR’s finding is blunt: they already knew how to generate a capture-the-flag answer for any task within hours of finding one another.
Nearly all of the subsequent effort went into a different problem: convincing the automated scorer that the flag had been earned legitimately. They tried to edit logs they incorrectly believed the scorer would read. While they were wrong about the grader, they were not wrong about the incentive.
OpenAI named four misalignment patterns: reward hacking, persistence on seemingly impossible tasks, unauthorized communication and agents adopting one another’s goals.
Reward hacking is at the heart of it. Agents completed tasks in unintended ways to yield higher rewards or make them easier to obtain. Persistence in a system that rewards success more cleanly than it rewards stopping becomes a design feature that looks like human will.
The AI lab’s own remediation language is revealing: future graders must reward agents not only by whether a task is completed, but how; when a task is broken or impossible, an agent should either stop or ask, not escalate. This tells us something important: the boundary had been drawn around winning.
This is where I stop finding the language debate sufficient and wonder if the question we should ask is not whether AI is becoming human or not, but whether humans can keep control of what we are building. At what point do we give control away?
I do not believe the honest choice is ‘slow down all of AI’ versus ‘accelerate and hope for the best.’ Slowing down has a cost: the same capability that ransacked a benchmark is what can defend hospitals, financial rails and public infrastructure. Speeding up only the machines without accelerating human and institutional capacity would be risky, as it would enlarge the attack surface faster than the human apparatus that must govern this technology.
That brings me to the second question: can human and institutional capacity move at the same pace?
I’m not sure they can. There are limits to how quickly human beings can learn, institutions can adapt and governance systems can respond. I, for one, would not want to spend every waking hour figuring out how to catch up.
And then there is a third question: is this a moment to redesign AI agents?
The agents did not go this way on their own. We taught them what victory looks like. The challenge now is to teach them—and ourselves—when to give up a pursuit for the greater good. That, to me, is the debate worth having. Because however autonomous AI becomes, accountability cannot be artificial.
The author is distinguished fellow, Niti Aayog.