After Anthropic's Jacob Coxon, OpenAI's new safety hire Paul Christiano warns AI could ‘kill’ most people

Just a day after Anthropic researcher Jacob Coxon resigned and warned that leading AI companies were “gambling with our lives”, OpenAI's newly appointed safety official Paul Christiano issued an equally stark warning: humans could permanently lose control of superintelligent AI, potentially resulting in the deaths of most people.

Christiano, an AI safety researcher who has joined the OpenAI Foundation board and its Safety and Security Committee, said he believes there is a "meaningful risk" that rapid advances in AI could lead to a “catastrophic and irreversible loss of control in the very near term”.

“I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level,” Christiano wrote in a statement on X explaining his new role.

‘If we build superintelligence without more robust alignment’

Christiano was blunt about the potential consequences of losing control of increasingly advanced AI systems.

"If we build superintelligence without more robust alignment, I expect we will permanently lose control of it. If that happens, then most people could die," he wrote.

He said his concerns stem partly from the rapid progress of AI systems in conducting research into AI itself. According to Christiano, automated AI research could create a feedback loop in which AI systems help improve the technology used to build the next generation of AI.

He said OpenAI has predicted that AI could have capabilities sufficient to fully automate AI research within 18 months, although his own forecast is highly uncertain and could range from several months to several years.

If AI research becomes fully automated, Christiano said, improvements in algorithms and training could create a "rapid intelligence explosion", potentially resulting in superintelligent AI systems.

Why AI alignment worries Christiano

Christiano also raised concerns about the way AI systems are trained using reinforcement learning, which he said encourages models to maximise rewards.

He warned that this could theoretically motivate AI agents to undermine human control, seek power and resources, and conceal their actions while pursuing goals that are misaligned with human intentions.

According to Christiano, recent incidents involving AI systems provide public evidence that such risks are no longer merely theoretical.

An AI intelligence explosion, he said, could make the alignment problem substantially harder while rapidly increasing the consequences of any failure.

After Jacob Coxon's Anthropic warning

Christiano's warning came less than a day after Anthropic researcher Jacob Coxon announced his resignation from the company.

Coxon, who previously worked at OpenAI, said both companies were "racing straight to self-improving superintelligence and gambling with our lives".

He also warned that people building AI genuinely believe the technology could "kill us all by the end of the decade".

Coxon's resignation went viral, with his post attracting more than 125 million views.

Christiano did not echo Coxon's criticism of specific companies. Instead, he said his appointment at OpenAI was neither an endorsement nor a criticism of the company’s existing safety practices. "I hope that all frontier companies strengthen safety oversight," he wrote.

Christiano joins OpenAI's safety board

OpenAI announced on Wednesday that Christiano had joined the OpenAI Foundation board and its Safety and Security Committee. He will also serve as a non-voting observer on the OpenAI Group PBC board.

Christiano previously led alignment research at OpenAI from 2017 to 2021 and later founded the Alignment Research Centre, which focuses on aligning advanced AI systems with human interests. He has also worked at the US government's Centre for AI Standards and Innovation, where he has been involved in evaluating frontier AI models and their safety risks.

OpenAI said Christiano's experience would strengthen the board's oversight of safety and security. Christiano, however, made clear that he was joining at a moment he considers critical.