The desperate search for AI’s most prized possession: the chip – 8/20/2023 – Tech

For the past 12 months, Jean Paoli, chief executive of Docugami, an artificial intelligence startup, has been looking nonstop for what has become tech’s hottest commodity: computer chips.

In particular, Paoli needs a type of chip known as a graphics processing unit, or GPU, because it’s the fastest and most efficient way to run the calculations that allow top artificial intelligence companies to analyze massive amounts of data.

So he reached out to everyone he knows in the industry who might be able to help. He applied for a government subsidy that allows access to the chips. He tried to make Docugami’s artificial intelligence technology more efficient, so as to require fewer GPUs. Two of his scientists even repurposed old video game chips to help.

“I think of them as a rare earth metal at the moment,” Paoli said of the chips.

More than money, engineering talent, media promotion or even profits, tech companies this year are desperate for GPUs. The search for the essential component began last year, when online chatbots like ChatGPT sparked a wave of enthusiasm about artificial intelligence, sending the entire tech industry rushing into the space and creating a shortage of chips. In response, startups and their investors are now scrambling to get their hands on the tiny bits of silicon and the crucial “computing power” they offer.

The shortage of artificial intelligence chips has been exacerbated as Nvidia, a veteran chip supplier, exerts virtual control over the market. Flooded by demand, the Silicon Valley company – whose market value has risen to a valuation of $1 trillion – is expected to report record financial results next week.

Tech companies typically buy access to artificial intelligence chips and the computing power they provide through cloud computing services from companies like Google, Microsoft and Amazon. That way, they don’t have to build and operate their own data centers filled with servers connected to specialized networking equipment.

But the explosion in artificial intelligence has created long waiting lists – reaching nearly a year in some cases – for access to these chips at cloud computing companies, creating an unexpected hurdle at a time when the technology sector sees nothing but opportunities and unlimited growth for companies that develop generative artificial intelligence, capable of creating its own images, texts and videos.

In general, larger technology companies are more easily able to obtain GPUs due to their size, vast resources, and market positions. This means that startups and researchers, who normally do not have this type of relationship or purchasing power, find it difficult.

An unforeseen wait

Their desperation is palpable. On social media, in blog posts, and at debates and conferences, startup founders and investors began sharing highly technical tips for tackling scarcity. Some are betting on how long they think it will take Nvidia’s waiting list to be filled. There’s even an awful song on YouTube, set to Billy Joel’s “We Didn’t Start the Fire,” in which an artist known as Weird AI Yankochip sings “GPUs are fire, we can never find them but we want to buy them.” them”.

Some venture capital firms, including Index Ventures, are now using their connections to buy chips and then offer them to companies they invest in. Entrepreneurs are bringing startups and research organizations together to buy and share a pool of GPUs.

At Docugami, Paoli evaluated the possibility of diverting GPU resources from research and development to the company’s product, an artificial intelligence service that analyzes documents. Two weeks ago, he struck gold: Docugami secured access to the computing power it needed through a government program called Access, run by the National Science Foundation (NSF), a federal agency that funds science and engineering. Docugami had previously received a subsidy from the NSF.

“That’s startup life when you need GPUs,” he said.

Who starts, suffers more

The lack of artificial intelligence chips has been most acute for companies that are just starting out. In June, Eric Jonas quit his job as a professor of computer science at the University of Chicago to raise money to start a drug discovery company using artificial intelligence. Scarce access to GPUs for university research projects was frustrating enough, but Jonas was shocked to discover that things were just as difficult for a start-up, he said.

“It’s the Wild West,” he said. “There is literally no capability.”

Jonas said he considered a number of undesirable options, including using older, less powerful chips and creating its own data center. He also considered using the chips from a friend’s Bitcoin mining rig — a computer designed to do the calculations that produce the digital currency — but thought that would be more work, as those chips weren’t programmed for the kind of work required by artificial intelligence.

For now, Jonas is calling in favors from friends at big equipment vendors and people who work at quantitative stock trading companies and may have extra GPUs or test labs that have GPUs he can use. He said he didn’t need much – just 64 GPUs for six hours at a time.

That tension is what recently led two business founders, Evan Conrad and Alex Gajewski, to create the San Francisco Compute Group, a project that plans to allow entrepreneurs and researchers to buy access to GPUs in small quantities. After exchanging hundreds of emails and a dozen phone calls with cloud computing companies, equipment manufacturers and brokers, they announced last month that they had secured 512 H100 chips from Nvidia and would rent them out to interested parties.

The ad gained “hilarious viral success,” Conrad said, and resulted in hundreds of messages from entrepreneurs, graduate students and other research organizations.

Conrad and Gajewski now plan to raise $25 million through a specialized type of bond that uses computer chips as collateral. The vendor, whose name the founders declined to mention for fear that someone would step in and buy the GPUs, promised access in about a month.

The pair said they hoped to help startups save money by buying only the computing power they needed to run experiments, rather than making large, years-long commitments.

“Otherwise, established companies win,” said Conrad.

free gpu

Equity investors for ventures have a similar goal. This month, Index Ventures partnered with Oracle to provide a combination of Nvidia’s H100 chips and an older version called the A100 to its very young portfolio companies at no cost.

Erin Price Wright, an investor at Index Ventures, said the company had seen its startups struggling to navigate the complicated process of getting compute capacity and signing up for waiting lists that reached nine months. Two companies have already decided to use the company’s new program, and others have expressed interest.

Before the shortage, George Sivulka, chief executive of Hebbia, a maker of artificial intelligence productivity software, would simply ask his cloud provider for more “instances,” or virtual servers packed with GPUs, as the company expanded. Now, he says, his contacts at cloud computing companies are either not responding to his requests or putting him on a four-month waiting list. He turned to customers and other connections to help make his case for cloud computing companies. And he is always looking for more.

“It’s almost like talking about drugs: ‘I know a guy who has H100s,'” he said.

Several months ago, some engineers at Hebbia set up a server with a few less efficient GPUs in the company’s Manhattan office, put the machine in a closet, and used it to work on smaller projects. Liquid cooling units keep the server from overheating, Sivulka said, but it is noisy.

“We keep the door closed,” he said. “No one sits next to him.”

Scarcity has created a stark contrast between the haves and the have-nots. In June, Inflection AI, an artificial intelligence startup in Palo Alto, Calif., announced that it had acquired 22,000 of Nvidia’s H100 chips. It also said it got $1.3 billion in funding from Microsoft, Nvidia and others. Mustafa Suleyman, chief executive of Inflection, said in an interview that the company planned to spend at least 95% of funds on GPUs.

“We’re talking about a seismic amount of computing,” he said. “It’s simply mouth-watering.”

Other startups have asked him to share, he said, but the company is already at full capacity.

_{Translated by Paulo Migliacci}