Reddit wants to get paid for helping teach AI – 4/19/2023 – Tech

Reddit has long been a popular space for internet conversations. About 57 million people visit the site every day to talk about a variety of topics, such as makeup, video games and tips for washing sidewalks.

In recent years, Reddit’s many chats have also been free learning aids for companies like Google, OpenAI, and Microsoft. They are using Reddit conversations to develop giant artificial intelligence systems that many in Silicon Valley think will be the tech industry’s next big thing.

Now Reddit wants to get paid for it. The company said on Tuesday that it intends to start charging big companies for access to its application programming interface, or API, the method by which outside parties can download and process the vast selection of conversations between individuals on the social network. .

“Reddit’s body of data is really valuable,” Steve Huffman, Reddit’s founder and chief executive, said in an interview. “But we don’t need to give all that value away for free to some of the biggest companies in the world.”

The measure marks one of the first significant examples of a social network charging for access to the conversations it hosts in order to develop AI systems such as ChatGPT, the popular OpenAI program. These new AI systems could one day lead to big business, but they probably won’t help companies like Reddit much. In fact, they can be used to create competitors – automated duplicates of Reddit conversations.

The Reddit move also comes as it prepares for a possible initial public offering on Wall Street later this year. The company, founded in 2005, makes most of its money from advertising and e-commerce transactions on its platform. Reddit said it is still working out the details of how much it will charge for API access and will announce pricing in the coming weeks.

Reddit’s conversation forums have become valuable commodities as large language models, or LLMs, are an essential part of creating a new AI technology.

LLMs are essentially sophisticated algorithms developed by companies like Google and OpenAI, which is a close partner of Microsoft. To the algorithms, Reddit conversations are data, which is among the vast pool of material that is fed into LLMs to develop them.

The underlying algorithm that helped build Bard, Google’s conversational AI service, is partially trained on Reddit data. OpenAI’s GPT Chat cites Reddit data as one of the sources of information it was trained on.

Other companies are also starting to see value in the conversations and images they host. Image hosting service Shutterstock also sold image data to OpenAI to help create Dall-E, a generative AI program that creates new, vivid graphic images with just a text command.

Last month, Twitter owner Elon Musk said he was cracking down on the use of the Twitter API, which is used by thousands of outside companies and independent developers to track the millions of conversations taking place on the network. While he didn’t cite LLMs as a reason for making the switch, the new fees could run into the tens or even hundreds of thousands of dollars.

To keep improving their models, artificial intelligence makers need two important things: an enormous amount of computing power and an enormous amount of data. Some of the biggest AI developers have a lot of computing power, but they still look outside their own networks for the data they need to improve their algorithms. This includes sources like Wikipedia, millions of digitized books, academic articles and Reddit.

Representatives for Google, Open AI and Microsoft did not immediately respond to requests for comment.

Reddit has long had a symbiotic relationship with the search engines of companies like Google and Microsoft. Search engines “crawl” Reddit pages to index information and make it available for search results. Such tracking, or “scraping”, is not always welcome on all websites. But Reddit benefited by appearing higher in search results.

The dynamic is different with LLMs – they devour as much data as they can to create new AI systems like chatbots.

Reddit believes its data is especially valuable because it is continually updated. It’s that novelty and relevance, Huffman said, that the algorithms in the big language models need to produce the best results.

“More than anywhere else on the internet, Reddit is a place for authentic conversation,” Huffman said. “There are a lot of things on the site that you would only say in therapy, or in AA, or never.”

Huffman said the Reddit API will still be free for developers who want to build apps that help people use Reddit. They could use the tools to create a bot that automatically checks that user comments follow the posting rules, for example. Researchers wishing to study Reddit data for academic or non-commercial purposes will continue to have free access to it.

Reddit also hopes to incorporate more so-called machine learning into the way the site itself operates. It can be used, for example, to identify the use of AI-generated text on Reddit and add a label that notifies users that the comment came from a bot.

The company also pledged to improve software tools that can be used by moderators — users who volunteer their time to keep the site’s forums running smoothly and improve conversations between users. And third-party bots that help moderators monitor forums will continue to be supported.

But for AI makers it’s time to pay.

“Crawling Reddit, generating value without returning any of that value to our users, is an issue for us,” Huffman said. “It’s a good time for us to adjust things.”

“We think it’s fair,” he added.

Translated by Luiz Roberto M. Gonçalves