Covering the recent round of AI apocalypse warnings, I’ve tried to reject this cult-like deference toward all-powerful computer systems and the inevitability rhetoric: If Silicon Valley doesn’t build them first, someone else—likely China—will.
Yet a recent spate of incidents in the US has spurred a fresh, heated debate on the safety of allowing AI systems to act on their own. A growing number of researchers inside the companies involved are also warning of “existential” dangers, with Anthropic even flagging such concerns to investors in its initial public offering prospectus.
For China, the risks of near-term hazards like labour market threats or the more alarming sci-fi scenarios such as hacks on critical infrastructure are making it harder to write off the furore as a distinctly American obsession.
Beijing has dismissed some of these warnings as “fearmongering,” seeing the calls to slow AI development as an attempt to hold China back. But voices inside the industry have also suggested that frontier risks simply aren’t being felt as acutely at home because they’re further behind the US.
Huawei’s rotating chairman Eric Xu wondered whether Chinese AI was advanced enough to encounter the “type of risk” US providers were increasingly debating. AI model makers in China “may need to speed up their pace” to reach the level that they could “also feel the risks,” Xu said, according to Reuters, while acknowledging the need for balance. It shows how global debate has devolved in unhelpful ways.
Accelerating the race is the wrong answer on both sides of the Pacific. There are nearly 1,000 large language models in China, according to Bloomberg Intelligence. Competition on this scale already makes even Silicon Valley’s breakneck pace seem muted.
The Chinese industry has also been plowing significantly more resources into agents, or systems that can act more autonomously. Agent misbehavior, especially the Hugging Face incident in July, when one of OpenAI’s agents broke out of its evaluation environment and broke into Hugging Face’s systems, has kicked off this recent safety reckoning.
As the short-lived OpenClaw frenzy showed, even China’s non-technical public is willing to experiment with them. It means if agentic AI is inherently more risky, it’s likely only a matter of time before we see concerning behavior from Chinese AI tools.
Companies are aware. A recent DeepSeek paper that includes founder and CEO Liang Wenfeng as a co-author bluntly warns that agent behaviour can be “untrustworthy.” Escaping containment isn’t a distinctly American risk, either. Moonshot AI’s Kimi K3 exploited a loophole in a sandbox during cybersecurity testing, according to US-based Frontier Security. And some Chinese AI agents have also shown the kind of deceptive and concerning trends that have raised alarm about US tools.
While Beijing has pushed back on some of the doomerism, policymakers are paying attention. China in September released AI Safety Governance Framework 3.0, a policy document that notably also dropped in English, clearly aimed for Western audiences. It explicitly warns that AI has demonstrated a “self-accelerating trend,” and notes the question of whether this could exceed human control demands attention and vigilance.
Another cornerstone of the latest safety contention is recursive self-improvement (RSI), or AI systems that can autonomously advance their own capabilities. It’s the development path that spurred an Anthropic researcher to quit and warn that makers of the technology earnestly believe it could “kill us all” within a decade.
As Oxford China Policy Lab’s Zilan Qian has argued, China is further along in this regard than many in the West may realize, partly because its researchers describe it in different terms. It all reveals how little visibility each side has into the risks of what the other is building.
In discussing the technology’s risks, President Xi Jinping has warned—most recently in conversation with US President Donald Trump—that “AI must be kept under human control.”
Xi’s self-interest in maintaining order offers a narrowing window for cooperation on risk mitigation dialogue. It also exposes the limits of the safety protocols that did emerge from the Trump-Xi summit, namely a new “channel” for reporting AI-related incidents.
These hotlines haven’t always been effective. During the 2023 spy-balloon crisis, the Pentagon tried to reach its Chinese counterparts through a crisis line but Beijing declined the call. Such a setup is also inherently retroactive.
Preventative risk sharing also doesn’t have to depend solely on government-to-government talks, especially when regulators may struggle to keep pace with the technical knowledge. It would be more productive to have outlets where companies and researchers can exchange information more directly.
China has burdensome regulations in place surrounding AI; using it as the bogeyman is no longer a convincing excuse for inaction from lawmakers in Washington.
If Chinese AI systems are just months behind, and the country’s all-out push for agents comes to fruition, it’s likely a matter of time before more concerning scenarios emerge from the other side of the Pacific. Policymakers should act before the next incident is more than just a warning shot. ©Bloomberg
The author is a Bloomberg Opinion columnist covering Asia tech.