The AI Boom Runs on Chips, but It Can’t Get Enough

Elon Musk of Tesla, SpaceX and Twitter talks with Thorold Barker of The Wall Street Journal about technology.
Elon Musk of Tesla, SpaceX and Twitter talks with Thorold Barker of The Wall Street Journal about technology.

Summary

  • ‘It’s like toilet paper during the pandemic.’ Startups, investors scrounge for computational firepower.

The artificial-intelligence revolution is being likened by Google’s chief executive to humanity’s harnessing of fire. Now if only the industry could secure the digital kindling to fuel it.

A shortage of the kind of advanced chips that are the lifeblood of new generative AI systems has set off a race to lock down computing power and find workarounds. The graphics chips, or GPUs, used for AI are almost all made by Nvidia. But the boom in demand for them has far outpaced supply with the viral success of ChatGPT, a chatbot that is able to respond to questions in humanlike ways.

“Because there is a shortage, it’s about who you know," said Sharon Zhou, co-founder and CEO of Lamini, a startup that helps companies build AI models like chatbots. “It’s like toilet paper during the pandemic."

That situation has restricted the processing power that cloud-service providers like Amazon.com and Microsoft can offer to clients such as OpenAI, the company behind ChatGPT. AI developers need the server capacity to develop and operate their increasingly complex models and help other companies build AI services.

Even the most connected tech entrepreneurs in the world are struggling to secure capacity. During a May 16 congressional hearing on AI, OpenAI CEO Sam Altman said it would be better if fewer people used ChatGPT because of the processor bottleneck.

“GPUs at this point are considerably harder to get than drugs," Elon Musk told The Wall Street Journal CEO Council Summit on May 23.

Being Musk has its perks, though. Earlier this year, startups clamoring for Oracle computing capacity were abruptly told that a buyer had snapped up much of Oracle’s spare server space, people familiar with the matter said. The buyer, the startups were told, was Musk, who is building his own OpenAI rival called X.AI, the people said.

Access to tens of thousands of advanced graphics chips is crucial for companies training large AI models that can generate original text and analysis. Without them, work on the large language models that are behind the AI runs much slower, founders say. Nvidia’s advanced graphic chips excel at doing lots of computations simultaneously, which is crucial for AI work.

UBS analysts estimate an earlier version of ChatGPT required about 10,000 graphic chips. Musk estimates that an updated version requires three to five times as many of Nvidia’s advanced processors.

Some investors are combing their networks for spare computing power while others are orchestrating bulk orders of processors and server capacity that can be shared across their AI startups. Startups are shrinking their AI models to make them more efficient, buying their own physical servers with relevant graphics chips or switching to less-popular cloud providers such as Oracle until the shortage is resolved, according to AI investors and startups.

Other founders are simply begging salespeople at Amazon and Microsoft for more power.

Zhou said Lamini, which she co-founded with a former Nvidia engineer, has the chips it needs. She and many other founders interviewed by the Journal declined to say precisely how they secured them.

“The industry is seeing strong demand for GPUs," an OpenAI spokesman said, adding the company is dedicated to ensuring its customers have the capacity they need.

Oracle and Musk didn’t respond to requests for comment. Microsoft and Amazon declined to comment.

Nvidia said recently that it is expanding its supplies to meet rising demand, but many AI founders expect the shortage to persist until at least next year. The demand for Nvidia’s products has driven the company’s stock up roughly 167%. Chip costs vary, but Nvidia’s advanced AI chips are sold by some retailers for around $33,000, though they can command higher prices in secondary markets amid the high demand.

Some companies are blocking off cloud capacity for fear they won’t be able to access them later. “People are now just continuing to pay for them even if they don’t need them," said Adam Wenchel, CEO of Arthur, which builds tools to protect companies from AI risks such as data leaks.

Companies that are able to secure computing power can still wait weeks to be able to use it, founders and investors say. “Even if you’ve already paid upfront, it doesn’t mean the GPUs come to you in the next day or week," said Aravind Srinivas, CEO of Perplexity AI, which builds an AI-driven conversational search tool. “You’ve just got to wait."

Server manufacturers and their direct customers say they are facing waits of more than six months to get Nvidia’s latest graphics chips. The CEO of Supermicro, one of the largest server-makers, said the company’s back-orders of systems featuring graphic chips was at its highest level ever and that the company was rushing to add manufacturing capacity.

All that has created a secondary market for these advanced chips, partly involving large crypto companies that bought chips during the boom to do mining and now don’t need amid a downturn in the digital-currency market.

Kanjun Qiu, the CEO of Generally Intelligent, an AI research company, has been buying advanced graphic chips since last year for her own servers, allowing her to ride out the current shortage. A venture capitalist recently messaged her asking if she had spare capacity that she could rent to other startups. Qiu hasn’t decided whether to part with her chips.

Meanwhile, OpenAI’s Altman and other employees have been fielding complaints from companies building AI services on top of their platform.

Alex Lebrun, the CEO and founder of Nabla, which developed an AI assistant for physicians, said OpenAI’s software can take up to two minutes to respond to queries. Nabla uses AI to automatically generate notes, referral letters and clinical recommendations, and its customers expect these notes to be produced instantly.

As a workaround, Nabla has built some simpler models to generate a first draft of the material more quickly, Lebrun said, and then relies on the latest version of ChatGPT for final adjustments. “The good startups are the ones that learn how to work around all these limitations," Lebrun said, adding that he has raised the problems with Altman directly.

Altman and other OpenAI representatives have told founders that the company is working to address the issue in partnership with Microsoft, its largest investor and data-center provider.

Catch all the Business News, Market News, Breaking News Events and Latest News Updates on Live Mint. Download The Mint News App to get Daily Market Updates.
more

topics

MINT SPECIALS

Switch to the Mint app for fast and personalized news - Get App