In a recent Financial Times column (2 July 2026, shorturl.at/y6B6Z), John Burn-Murdoch and Sarah O’Connor set out, lucidly and a little chillingly, the asymmetric risks of the AI revolution.
Burn-Murdoch’s observation is that the threshold for danger sits far below the threshold for usefulness. For AI to pose a large-scale danger, it needs only to have a reasonable chance—one in two, say—of succeeding at the kind of software tasks that enable systems hacking or cyber-attacks, whose consequences are physical as much as digital.
By contrast, to replace human workers, it must be reliable and consistent close to 100% of the time across a wide range of tasks that extend well beyond software into the inherent messiness of dealing with humans and navigating the physical world.
O’Connor draws the disquieting corollary: we may have built a technology already capable of being dangerous but not yet reliable enough to be useful in many domains. If so, she asks, should the incentives for AI labs not turn towards reliability, given the enterprise adoption it would unlock, rather than pushing the capability charts ever higher?
It reminded me of what Mustafa Suleyman wrote in The Coming Wave three years ago. Discussing the risks of AI, he turned to gain-of-function (GoF) research—ostensibly meant to prepare us for deadlier pathogens—to make a point that has stayed with me: the real danger is not bad actors weaponizing technology, but the unintended harm that flows from good people trying to improve health outcomes.
The problem, I posit, is a function of ego and its egregious cousin, hubris. Ego is the biggest driver of human ingenuity, innovation and progress. And the very ingenuity it drives, paradoxically, arms bad actors too, for the instruments of progress are as available to those with ill intent as to anyone else.
The answer would seem to lie in accepting humankind’s limitations, stopping somewhere and not progressing further. But the notion of halting progress is anathema to many. So, the logical, though not the most desirable, conclusion seems to be that we are fated to remain this way and continue to destroy ourselves.
This is pessimistic. Let us try harder.
A possible answer may lie in every human being deciding when and where to stop. Each one knows deep down when they cross the threshold from intellectual curiosity or a quest for progress to pursuits driven by delusions of grandeur.
Another possible answer is that a disaster, or one narrowly averted, awakens leaders the world over to the need for a consensus on responsible and sustainable innovation.
Immediately, the mind recalls several such instances of mini-disasters that thankfully did not turn into major disasters, as recounted by Suleyman in his book; yet we have marched on, disregarding them. So, this avenue does not seem promising either.
Though it is difficult and may even seem impossible, a fruitful way out could be to remind humans of their ability to figure out when to stop. Recall Peter’s Principle: in any hierarchy, every employee rises to his ultimate level of incompetence. Therefore, the key to a life that one can look back on in dotage with quiet satisfaction is to stop at a level below ultimate incompetence.
Or, take a successful athlete. The one who chooses to leave at his peak is applauded for that act of sagacity and wisdom, and remembered. Take investing. The secret to growing or preserving one’s wealth is knowing when to quit and when to cut one’s losses.
But notice that in each of these examples, the downside of not stopping is borne by the individual—the athlete, the investor or the employee. The trouble with AI, and with gain-of-function research, is that the incentives are upside down. The one who chooses not to stop keeps the glory, the patent, the citation; the cost of this person not stopping is borne by others who cannot easily trace the harm back to him or her.
It is easy to embrace restraint when it is rewarding. If restraint robs the practitioner of the upside, then asking this person to do so in the interests of society is harder. This is why individual wisdom, though necessary, cannot by itself be enough.
We appear to have come full circle, with little to show for it. And yet the circle is not quite closed. If individuals cannot be relied upon to act against their own interests, then the question is whether incentives will work.
Here, O’Connor’s instinct is sound, but her question is not. She has seen the asymmetry clearly, but has chosen between reliability and raw capability. The pressing challenge is about lengthening the odds of AI causing harm or of malicious actors using it to cause harm.
Driving the success rate of useful tasks towards 100% does nothing to drive the success rate of dangerous ones towards zero. Commercial incentives will take care of the former, whereas the latter is a deliberate choice.
The question is what will make AI companies take the risk of harm more seriously than passing capability benchmarks, which themselves are not quite what they seem, being defined narrowly enough that a model can top them by memorizing or gaming the test rather than by acquiring the capability that is being certified (International AI Safety Report 2026, shorturl.at/hcpKy).
Will it be law or markets? The race to maximize capability for civil and military applications makes governments wary of pursuing safety. Markets do not reward those who mind risks, for markets have all but forgotten how to recognize risk, let alone price it, ever since easy money in the new millennium made it easier for investors to chase returns, ignoring risk.
So, the one who presses on keeps the upside; when the reckoning comes, no one will be held to account, as happened in 2008. By then, the accounting will not matter.
These are the author’s personal views.
The author is chief economic advisor, Government of India.