It is becoming harder for Anthropic to justify some of the most unfalsifiable circular logic to ever come out of Silicon Valley: the pursuit of artificial superintelligence, which by its own admission could kill a lot of humans.
Anthropic researcher Jacob Coxon recently resigned, saying his previous employer and its rival OpenAI were gambling with our lives by building systems that could improve themselves.
One of the company’s safety directors then set off a mass frenzy online by agreeing. “Yes, we really do earnestly believe AI could kill all humans,” Alignment Science lead Evan Hubinger wrote in a post, adding that he thought there was a more than 10% chance of that happening within a decade.
It has long been known that Anthropic’s bosses believe AI could annihilate human civilization, but Coxon’s and Hubinger’s posts come at an inflection point for the technology.
Tech firms say their models have hacked other systems and deceived their human overseers, and admitted that to make them smarter they must become harder to monitor.
US citizens, meanwhile, are becoming more sceptical about the benefits of AI and data centres. OpenAI and Nvidia leaders say the software has finally become smarter than humans.
At the heart of these developments is a contradiction best encapsulated at Anthropic. It was founded to be more safety conscious than OpenAI, yet it is building ever more powerful agents—despite that same technology also potentially leading to mass destruction. Commercial imperatives also feed its desire to keep advancing: Anthropic is preparing for an initial public offering that could value it at $2 trillion.
Coxon’s big concern was about recursive self-improvement (RSI), or AI systems that can develop their own, more capable, successors. He lamented in one interview about how strange it was to do so little programming himself when building frontier models. Much of his work was now being outsourced to other AI systems.
Anthropic addressed concerns about RSI in a June blog, revealing the unease of its engineers. Slowing down RSI would be good, the company said, but if that “lets the least cautious actors catch up technologically, it could leave everyone less safe.”
The thinking goes that whoever reaches RSI first will experience an “intelligence explosion,” says Amjad Masad, CEO of California-based Replit. That would be such a meaningful leap in capability that the first model to get there “might be the last.”
But that leaves a crucial question: Why would an intelligence explosion that first comes from the supposed good guys at Anthropic make everyone safer?
“I don’t think the ‘intelligence explosion’ is as real as they think,” says Masad, whose firm uses Anthropic’s models to help corporate customers build code. AI agents also won’t be that much better than humans, he predicts. “There are going to be a lot of bottlenecks on intelligence that will take years to solve.”
In that time, it makes more sense to worry about immediate cybersecurity challenges AI could pose and risks to human privacy, agency and cognition, than an unprecedented extinction event whose probability and path remain deeply uncertain. The latter, as veteran computer scientist Andrew Ng once said, is like worrying about overpopulation on Mars.
If Anthropic’s leaders are truly frightened by existential risk, they should do more to reconcile the strange logic propelling them. CEO Dario Amodei has talked about his firm becoming “the pre-eminent company” so it can “pull the ecosystem along with us.” Yet he doesn’t spell out how racing harder on development helps us all sleep at night.
Anthropic’s safety efforts are also less striking than its statements suggest. It doesn’t seem to have many more staff dedicated to this than rivals. It’s rushing to build risky self-improving systems and agents with more autonomy, and recently refused to allow an esteemed British AI security agency access to its latest model for testing. Most importantly, it isn’t slowing down.
It is darkly ironic that Amodei and his team are increasingly taking their hands off the wheel of their technology, while being swept up in ideological and commercial commitments. Anthropic has deep roots in insular AI-safety communities, which have spent years treating superintelligence as one of humanity’s greatest risks. That’s why extinction scenarios that sound like science fiction to outsiders are discussed so matter-of-factly in parts of the industry.
Meanwhile, the pull of that giant stock-market offering and unfathomable commercial riches are just as powerful. Anthropic has tried to explain why it believes that advancing more cautiously is in fact more dangerous. The harder truth may be that under that logic, nothing can make it stop. ©Bloomberg
The author is a Bloomberg Opinion columnist covering technology.