Video: startup promises chip that runs AI in milliseconds – 02/23/2024 – Tech

The American startup Groq created a chip capable of executing commands in ChatGPT in 7% of the time it takes in the Microsoft cloud, at a speed 13 times faster.

Videos of the Groq platform responding to requests in a fraction of a second are circulating on social media.

The company says it delivers answers with the highest speed in the world, and platforms specialized in testing the performance of AI models endorse it.

Groq makes chips just like Nvidia and is not Elon Musk’s unmoderated AI model called Grok. The name of the two companies, which means deep and intuitive knowledge, originates from the science fiction book “A Stranger in a Strange Land”.

The startup’s proposal, however, is a different architecture from Nvdia’s GPUs (graphics processing units), known for allowing the training of large language models such as ChatGPT. Groq says it produces LPUs (language processing units).

The company’s project says it will abandon everything that is not necessary to run an AI platform and save more space for parallel electronic circuits, essential for carrying out the statistical calculations involved in executing a language model.

In this way, Groq can make an AI model respond at a speed of 246 tokens per second — tokens are the codes from which the technology processes words, each one represents a piece of word. This, according to measurements from the AI ​​benchmarking company Artificial Analysis.

Microsoft provider Azure, for example, makes ChatGPT run at a speed of 18 tokens per second.

Groq’s chips cannot yet be produced on a commercial scale, like Nvidia’s GPUs. The startup projects that this will be possible in 2025.

Getting faster responses from an AI model can open up space for new applications.

An example of this is the current speed of ChatGPT, which is currently incompatible with human dialogue, which makes the experience robotic.

In an interview with CNN Internacional, Groq’s chief executive, Jonathan Ross, showed that his company’s chip makes a casual conversation possible. The chatbot maintained a simple dialogue with the host of the Connect program, Becky Anderson.

Groq, however, is not the only company working on inference chips aimed at artificial intelligence. Nvidia, Google and Microsoft have already released similar projects without full public access.

Groq’s LPUs are also not as efficient as Nvidia’s GPUs at training large language models. This is the essential step to make platforms like ChatGPT deliver adequate responses.

On February 8, the Wall Street Journal reported that Sam Altman, the chief executive behind ChatGPT, was moving to raise up to US$7 trillion (R$34.6 trillion) to overhaul chip production around the world. This deal, however, is still limited to a rumor.