Musk and others want AI tools paused but it’s too late for that

Elon Musk and an array of public figures signed an open letter that went viral last week, calling for a six-month pause on training language models more powerful than GPT-4, the tech behind ChatGPT. The letter is odd. It criticizes the deployment of powerful chatbot technology as rash, but also over-hypes their capabilities, drawing on AI doom-mongering that has captivated some of the press and distracted us from more nuanced risks. “Should we develop non-human minds that might eventually outnumber, outsmart, obsolete and replace us?” the letter asks. “Should we risk loss of control of our civilization?”

Of course not, but there are issues we should be more worried about now, like the concentration of AI capabilities among a few increasingly secretive companies, inequality as artists find their work plagiarized without compensation, and all the risks to come from companies racing to plug ChatGPT into their systems.

On that last point, toothpaste is already out of the tube. OpenAI last week launched a new system that will allow businesses to plug ChatGPT into their proprietary databases, allowing its chatbot to carry out tasks on their systems like retrieving information, making bookings and even running new software that it creates.

While the plug-in announcement didn’t get much attention in the mainstream press, many technologists saw it as a stunning leap forward for ChatGPT. Not only could it search and synthetize information it had been trained on, it could take action.

Think about that for a moment. Machine learning (ML) systems make decisions in an inscrutable black box. OpenAI spent seven months testing GPT-4 before releasing it into to the wild, but its so-called ‘red team’ engineers, who tested how it might be misused, could only cover a fraction of the ways it might be exploited by millions of real-world users. However much OpenAI has tested and prodded its system to make sure it is safe, no one really knows the full extent of its risks until it is deployed publicly. And those risks become more serious when ChatGPT can start doing things on the web.

Taking such gambles is, of course, what made ChatGPT so popular. OpenAI made a bet by making ChatGPT public and the result was public adulation and viral success for an AI team that’s tiny compared with those that work for Big Tech firms.

Now with these new plug-ins, OpenAI is taking another gamble, giving ChatGPT even more capabilities and unleashing it to businesses. So far, 11 companies including Expedia Group and payments operator Klarna Bank have ‘plugged’ ChatGPT into their servers as initial test cases.

Some of the new capabilities that OpenAI is offering businesses look dicey. One, called ‘code interpreter’, allows a business to get ChatGPT to write and even execute code on its behalf. Software developers typically use a code interpreter to make sure the program they’re writing works as intended, allowing them to test small snippets. With the new plug-in, ChatGPT can also write and test code, and that sounds potentially hazardous from a security perspective.

“I hope that there are enough guardrails in place,” said Edo Liberty, the former head of Amazon AI Labs and current CEO of Pinecone, a startup that has a technical partnership with OpenAI on one of its new plug-ins, when I asked him about the risks of letting ChatGPT run code. “This is something that could theoretically be a vulnerability.”

Marco Cardoso, a software engineer with Microsoft based in Sao Paulo Brazil, tested ChatGPT’s new code-writing abilities and noted how remarkable it was that the system could “now make requests to any server on the Internet.” But that also meant there was “nothing stopping” ChatGPT from accessing a network it wasn’t supposed to. To prevent that, Cardoso put in his own guardrail, explicitly instructing the tool to only access one particular domain. But that puts extra responsibility on the users of ChatGPT, he noted. What’s to stop it from making a request to an unknown server?

Ethan Mollick, a professor at Wharton who has experimented extensively with ChatGPT, tweeted this week that trying out its plug-ins put a spotlight on OpenAI’s own research about the “unpredictable risks” of connecting the tool to other systems.

OpenAI’s nimbleness as a small company has allowed it to move quickly to release remarkable new technology to the public, from DALL-E 2 to ChatGPT to GPT-4, all within a year, achieving the kind of impact that much larger companies could only dream of. But the flip side is that we’re all guinea pigs for utterly new and potentially flawed technology.

In some ways, it doesn’t really matter if OpenAI agrees to the open letter and pauses its development work for six months. With companies rushing to plug ChatGPT into their systems and test it out on the public, it may already be too late.

Parmy Olson is a Bloomberg Opinion columnist covering technology.