It could kill us all by the end of the decade’: Anthropic researcher quits, warns AI firms are ‘gambling with our lives’

A researcher associated with Tech Giant Anthropic was quitting the industry allegedly over fears that the company and its competitors were 'racing to build systems they won’t be able to control'.

What Jacob Coxon said

Jacob Coxon, whose X profile describes him as an AI researcher, also wrote a series of posts on X where he detailed out the reasons for his purported resignation from Anthropic.

"I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," he wrote

According to Coxon the systems could soon have superhuman strength and this poses a level of danger to everyone.

"Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing," he posted.

"The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger," he added.

Coxon alleges that companies like Anthropic understand the stakes but are involved in a race to get to the end result first.

"A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk," he posted.

"Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available," he added.

Coxon advocates for extreme measures including a possible ban on improving AI model capabilities.

"I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities," he said.

"If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a super-intelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions," he added.

‘AI could kill all humans’

Responding to him, head of alignment at Anthrophic, Evan Hubinger said:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

Earlier on August 16, Anthropic CEO Dario Amodei had pushed back against claims that regulating artificial intelligence would inevitably concentrate power in the hands of a few companies and governments, arguing that carefully designed rules could instead constrain frontier AI firms while giving smaller competitors more room to catch up.

Responding to a debate sparked by comments from investor Gavin Baker, Amodei said the choice between concentrating AI power through regulation and distributing it widely was a "false choice". He argued that fair institutional processes can constrain corporate power and protect individuals, while regulation can be designed to place greater burdens on the most advanced AI companies than on smaller challengers.

The comments came in response to Baker's argument that AI's risks should potentially be addressed by distributing AI capabilities widely rather than concentrating them through regulation.