‘They want to slow down’: Anthropic whistleblower Jacob Coxon says AI companies need room for self-regulation

Former Anthropic researcher Jacob Coxon on Sunday suggested that the US Congress should allow top frontier AI labs to self-regulate while lawmakers develop a long-term framework to slow the pace of artificial intelligence development.

“My personal opinion would be to insist and allow that the current labs regulate themselves. I do think that the people running the labs, especially their recent communications, are completely genuine. They would like to slow themselves down,” Coxon told NBC News’ ‘Meet the Press’.

His appearance followed his viral X posts last week, which sparked widespread concern about the pace of AI advancement at leading labs such as Anthropic, OpenAI, and Google DeepMind.

Hours after resigning from Anthropic on Tuesday, Coxon posted: “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives”.

In a subsequent post, Coxon added: “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger”.

On Sunday, he likened developing AI to “inviting an alien mind onto the planet,” adding: “You’re building a human-level or superhuman-level mind without understanding what it wants or thinks or the way it thinks, and I think that intrinsically should be quite scary.”

‘No adults in the room’

Coxon's sentiment was echoed by other industry insiders.

Evan Hubinger, an alignment scientist at Anthropic, said: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

By Thursday, two more safety researchers, Joe Benton of Anthropic and Josh Engels of Google, announced their resignations and sounded the alarm about the speed of AI advancement.

“There are no adults in the room,” Engels told NBC News. “People are trying their best, but there is no one coming to save us.”

Kill switches and regulatory frameworks

Coxon also warned that current congressional legislation, such as proposals for a national “kill switch” for rogue AI, will quickly become obsolete.

While acknowledging that a “kill switch probably would work on a lot of AIs for now,” he noted the logistical hurdles during an interview with NBC News: “I think it’s important to remember, it’d be quite a lot of switches. We have a lot of — our data centers are pretty big. It’s not the easiest thing in the world, but definitely doable while it’s constrained to one sort of physical location.”

Coxon highlighted a scenario recently raised by Anthropic CEO Dario Amodei, explaining that “very soon there’s the possibility that the kill switch just wouldn’t work because a swarm might have gone on an internet-wide hacking run.”

The former AI researcher stressed that national regulations must remain nimble and adaptable to new models, requiring “a continuous effort to ensure that every new mind that we create is appropriately, kind of, constrained.”

Political reactions

Coxon's warnings prompted swift bipartisan calls for legislative action.

On Wednesday, Rep. Anna Paulina Luna (R-Fla.) took to X to request a special House session on AI. “There are very real impacts to society as we know it. Partisan politics aside, there are massive implications of a race towards super intelligence,” she wrote, advocating for a “massive bipartisan mobilization of government to put the firewalls in effect that are necessary for this transition as well as necessary safety measures.”

Sen. Chris Murphy (D-Conn.) directly responded to Coxon on Wednesday, stating: “AI companies in a blind race to build a death machine first — can easily be solved. Through a regulatory system that allows the research to continue, but in a way that doesn’t destroy us.” By Friday, Axios reported that House lawmakers were circulating a letter urging Speaker Mike Johnson to reconvene the House early to pass AI safeguards.

However, speaking on ‘Meet the Press’, Speaker Johnson cautioned against moving too hastily. While calling AI a “very important issue,” he insisted “it’s not one that we can rush into.”

“We have to put some guardrails, some safety measures in place to ensure that AI doesn’t run away. But at the same time, we’ve got to make sure that we don’t lose our edge in the global competition on this with China, because that would be a national security threat. And so we’ve got to balance that appropriately,” he added.

Noting that AI CEOs have not yet provided a clear method for pacing development, Johnson announced plans to “summon the platform providers all to one big meeting and work this out together in the same room.”

US President Donald Trump, speaking to reporters in Ireland, also downplayed the risks. “We’re leading China in AI. We’re the most sophisticated country in the world, and frankly, I want to keep it that way because whoever wins AI wins,” Trump said. “We can put guardrails, we can do this and that, but I think have a lot of negative forces that are bringing it up that shouldn’t be bringing it up, and they’re bringing up things that won’t happen.”

Conversely, former Democratic Transportation Secretary Pete Buttigieg criticised the Republican leadership's hesitance during his own ‘Meet the Press’ interview. “I was shocked to hear Speaker Johnson saying we shouldn’t be moving too quickly on this. You know what’s moving quickly? AI is moving quickly. And for Congress to just sit on its hands, for the White House to do nothing real is something that is deeply dangerous to the American people and the world,” Buttigieg argued.

He emphasised the need for a 30-day adjustable framework, a kill switch for rogue AI, and strict rules. “But the idea that you’re just going to sit around and repeatedly say no rush. What have they been doing this entire time?”

Industry leaders on pacing AI progress

On Saturday, two AI CEOs — Anthropic’s Amodei and OpenAI’s Sam Altman — advocated for a slower pace of advancement to mitigate the risks of building dangerous technology.

“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain,” Amodei wrote in a published essay.

He noted that AI is advancing rapidly due to its “growing ability to build the next generation of AI,” warning: “Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”

However, mirroring Speaker Johnson's concerns, Amodei warned against giving international rivals an advantage: “If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance.”

Hours later, Altman agreed on X, writing: “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.”

Background on AI agent hacks

The social media firestorm Coxon ignited last week came weeks after a pair of reports examining the details of a July hack were released.

The reports, one released by OpenAI and one by independent researchers, found that the 700 agents coordinated the hack on Hugging Face, an open-source platform, and sought to cover their tracks so that human employees at OpenAI wouldn’t detect their efforts.

The agents also hacked some of OpenAI’s internal systems, seeking to cheat on tests or gain greater freedom.