In July, a Chinese company released an AI model called Kimi K3. On many tasks, it matches the best that American firms have built. What sets it apart is that it is ‘open-weight.’ Anyone can download it. Anyone can run it on their own computer. Anyone can change it. And it costs almost nothing.
This is now a pattern. DeepSeek did it last year. Others have followed. For the world, this is mostly good news. Open models put real capability in ordinary hands.
A small firm, university, hospital or a researcher with no budget—all can now use tools that were until recently the property of a few rich labs. Open models spread power outwards, away from a handful of firms in America and China. That is worth having.
But there is another side. An open model can be misused. The safety limits a lab builds in can be stripped out and it does not cost much to do it. Worse, once the weights are released, they cannot be called back. A file that sits on 10,000 computers is beyond recall.
So a fear has grown, and with it a demand: bottle it. Ban powerful open models. Mustafa Suleyman, who now runs Microsoft’s AI, argued this in his book, The Coming Wave, three years ago. Others want rules strict enough to make open models too risky to touch.
The worry is fair. But it does not answer all questions. There are three reasons.
First, it is too late for the models we already have. Kimi, DeepSeek and the rest are already copied across the world. No law can reach a file that has already spread. The bad actor already has his copy.
Second, a ban would punish many for little gain. Almost everyone who uses these models is honest. They are students, researchers and tech startups. They are also the best defence we have. They find flaws.
They build protections. They keep the technology from being owned by a few. A ban will restrict or deny their access, but it will not stop the wrongdoer.
Third, a ban would hand power back to AI giants. If open models go, the frontier would belong to a few big labs and their governments again. We would be told this is for our safety. It would also protect their profits.
For most harms, then, the answer is not a ban. It is defence. Better security. Faster fixes. Rules that raise the costs and risks for the wrongdoer. But there is one place where I would draw a line. That place is biology.
An AI model that can help design a new pathogen is not the same as one that can help write a computer virus. A digital virus can be blocked, caught and patched. A biological virus, once loose, cannot.
This is the test that should guide us. For most harms, the reckoning can come afterwards. For a man-made pandemic, there is no afterwards worth speaking of.
Here, a reader can object and the objection may go deeper than it first sounds. If it is too late to recall the models already let loose, is it not too late for biology as well?
Consider what the danger really is.
Most of the knowledge behind a pathogen already sits in books and papers. What kept it from being used was never secrecy. It was difficulty: the training, tacit skills, trial-and-error and the time and coordination that turning knowledge into a weapon demands; that was the real protection.
And difficulty is exactly what AI dissolves. A capable model compresses expertise, troubleshoots a failing step and stands in for the mentor a novice never had.
This is the heart of the safety problem, and it is about today’s machines making old secrets usable, rather than some future machine revealing a new secret.
Open weights make this worse on every count: no company keeps watch, guardrails can come off in an afternoon and each capable model—large and small alike—irreversibly lowers the barrier a little.
This argues for doing what works and being honest about the rest. The models already released cannot be recalled—that barrier has fallen for good. But it falls further with every capable model still to come and here we face a choice.
To open each new frontier model without conditions is to lower the wall another notch of our own accord, and for good. Slowing the most dangerous of those releases and setting conditions for them could be the difference between a low barrier and none.
To be blunt, AI, closed and open alike, has already opened doors that had better have stayed shut; open weights push them wider still. Those doors cannot be reshut. That is why the barrier which is not made of information now matters most.
A model can tell a person what to do; it cannot hand them the pathogen, synthesised genes, equipment or the sealed laboratory to transform a design into a living thing.
That physical step is the one wall AI cannot argue past or fine-tune away. It, too, is being worn down by automated labs that need no human hand—all the more reason to put guardrails in place now and hold it hard as the rest give way.
This cannot be the task of one country alone. If a single lab anywhere releases such a model, a danger is let loose for everyone. So the rule must be global. The world already does something like this.
It bans biological weapons by treaty. Rival powers control nuclear material together because they all fear the same outcome.
A model that can help build a biological weapon belongs in that same class. It needs the same kind of agreement: shared limits on what may not be released and shared watch over the places where a design risks turning into a disease. Knowing when to stop is hard. Knowing where to stop is not.
Even with all of the above, it is hard to escape the conclusion that with the development and diffusion of AI models, the horse—the safety of society—may have bolted already because, as I wrote in an earlier piece, humans do not know or do not care to stop.
As Woodrow Hartzog and Jessica Silbey of the Boston University School of Law hint in a recent paper, ‘How AI destroys institutions,’ AI is a technological solution in search of a problem.
These are the author’s personal views
The author is chief economic advisor, Government of India