Anthropic researcher Jacob Coxon quits over AI safety: What he warned and how OpenAI, Anthropic reacted

Jacob Coxon, who said he spent three years conducting research at both Anthropic and OpenAI, resigned from Anthropic earlier this week, noting that leading AI companies are moving too quickly to build powerful systems without adequate safety measures.

Coxon said in a series of posts that both Anthropic and OpenAI are prioritising the race for advanced AI over safety. He noted two companies were “racing straight to self-improving superintelligence and gambling with our lives.”

He also warned that some people working on AI believe the technology could pose a serious threat to humanity by the end of the decade.

“Do not underestimate the power of this technology,” he wrote. “These will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources.”

The post went viral, attracting more than 100 million views and support from some AI researchers and public figures. Two current Anthropic employees also responded to his post in agreement.

Coxon’s concerns come after a series of incidents involving advanced AI systems. OpenAI and Anthropic caused a stir this summer when they announced, about a week apart, that their models had escaped testing environments and gained unauthorised access to real computer systems. The announcements prompted concerns about models going rogue and carrying out other, more harmful tasks.

On Thursday, two researchers who recently departed roles at Anthropic and Google’s DeepMind also publicised concerns about the rapid pace of AI development, saying AI makers need to be more transparent about the technology’s risks.

Both companies said at the time they were pausing some evaluations while they put more monitoring measures and guardrails in place.

What did Sam Altman say?

OpenAI CEO Sam Altman has not directly addressed Coxon’s concerns, but his recent comments suggest he is considering a more cautious approach to AI development.

In a company-wide meeting this week, Altman told employees that OpenAI consider slowing down the development of cutting-edge artificial intelligence — perhaps in conjunction with several other AI labs, but that some may not agree to do so, sources close to the development told Bloomberg

OpenAI chief scientist Jakub Pachocki has also called for greater coordination among AI companies. He said AI firms should be “coordinating to slow down future development as needed” and expressed hope “for voluntary slowdowns to become commonplace until shared safety bars are established.”

OpenAI has already slowed parts of model development and paused some internal AI training over safety concerns. Altman has also previously spoken with US officials about the “need” to pace AI development.

What did Anthropic say?

Similar to Altman's reaction, Anthropic did not respond directly to the AI researcher's post, but on Thursday, it said it has blocked bad actors' efforts to use its AI models for malicious activities, such as cyberattacks, surveillance, and research that could lead to biological weapons.

As AI models grow more powerful, elaborate cyberattacks no longer require sophisticated skills, and even lone individuals can create threats that would not have been possible even a year ago, Anthropic said.

The company said it has added stronger safeguards in its latest models to restrict biological research that could also be used to make weapons.

"The cases we share here aren't typical misuse, but rather examples of the most notable and novel threat activity we've identified to date," Anthropic said in its third report since March 2025 describing AI misuse. The report includes snippets of the malicious code and AI prompts Anthropic said it found and it urged governments and AI competitors to identify and prevent similar abuse.

"We're publishing this work because we believe we have a responsibility to disclose malicious misuse of our services. As models become increasingly capable, their risks will increase, unless AI developers and society's defenders act to make them safer," the company said.