Search wars reignited by artificial intelligence breakthroughs

The internet search wars are back.

The emergence of AI systems capable of generating direct textual answers to questions — most notably the ChatGPT chatbot created by San Francisco-based OpenAI — has opened the first new front in the battle for search dominance since Google fended off a concerted challenge from Microsoft’s Bing more than a decade ago.

Google and Microsoft are both close to announcing revamps of their search engines to include direct answers supplied by artificial intelligence. Meanwhile, several search start-ups have already embedded AI in their services, giving the first glimpse of how the technology behind ChatGPT might transform one of the biggest online markets.

The sudden spurt of experimentation is long overdue, said Greg Sterling, an analyst who has followed the search market since 1999. For younger users in particular, Google’s search results pages seem cluttered and strewn with advertising, he said. “People are ready for something that is simpler, seemingly more credible and doesn’t have tons of ads stuffed in it.”

On their own, systems such as ChatGPT, based on so-called large language models that can “understand” complex queries and generate text responses, do not represent a direct alternative to search. The information used to train ChatGPT is at least a year old and the answers it gives are limited to information already in its “memory”, rather than more targeted material pulled from the web in response to specific queries.

That has led to a race to develop a new hybrid of AI and traditional search. Known as retrieval augmented generation, the technique involves first applying search tools to identify the pages with the most relevant material, then using natural language processing to “read” them. The results are injected into a large language model such as OpenAI’s GPT-3, which then spits out a more precise answer.

Google’s rivals say this has led to a rare opportunity to create new and distinctive search services, rather than trying to beat the search giant at its own game.

“Google has gotten away with it because until now everyone, including us, has tried to beat them with a better set of links,” said Sridhar Ramaswamy, a former Google executive and co-founder of search start-up Neeva.

His company has already begun inserting short textual answers into its search results pages. In an effort to overcome an inherent weakness in large language models — that they sometimes produce incorrect results — Neeva was also the first to add citations to its textual answers, showing the sources behind its assertions.

Meanwhile, some companies have opted for the more radical approach of presenting users with a pure chat interface that is similar to ChatGPT, dispensing with search results pages altogether.

Microsoft is planning to let users “toggle” between a chat interface and a traditional search results page, according to a number of people who saw what appeared to be an early release of its revamped Bing service late last week. The software company refused to confirm that screenshots of the apparent online leak were genuine. You.com, another search start-up, has already taken this approach, adding a “chat” button for users to switch out of standard search.

A pure chat service is much more appealing to a younger generation that wants a completely different experience, said Angela Hoover, co-founder of Andi, another search start-up.

“Gen Z is turning to TikTok because they’re so hungry for something new,” she said. “They want factually accurate answers that can be presented to them visually and in a conversational way.”

Aravind Srinivas, co-founder of Perplexity AI, said that going all-in on the conversational interface would also lead to a more distinctive and useful service. “This is basically Wikipedia on steroids, it will transcend notions of search engines,” he said.

However, companies that use AI to supplement their search results may find any advantage shortlived. Google said last week that it would start to include results from its own language AI systems in its results “very soon”, and that it would let users interact with the new AI technology directly.

That may make it hard for rivals to come up with anything distinctive enough to break Google’s grip on the market, said Sterling. “If Bing integrates AI into search in the same way Google does then we might not see any change,” he added.

Yet even if the impact on competition turns out to be muted, the broader economic implications for the web still look profound. For publishers who rely on web traffic generated by search engines, for instance, there is a risk that internet users will find answers without needing to click through to the underlying pages.

“Referral traffic will definitely drop,” said Srinivas at Perplexity. “This is not great for publishers.”

He predicted this would force publishers to rethink their relationship with search engines, leading them either to wall off their sites or try to negotiate a fee for allowing their information to be crawled and processed.

For the search engines, meanwhile, the higher cost of generating full-text answers threatens to jeopardise what until now has been a high-margin business. Srinivas estimated that a search involving AI could cost seven or eight times more than a traditional internet search.

“These things can only succeed if the value per query is much higher,” he said.

For Microsoft, which spent billions of dollars in the early days of Bing trying to compete with Google, lower profit margins may be an acceptable cost for the chance to finally win significant market share. Driving down profit margins for everyone might also serve a broader strategic purpose if it helped Microsoft weaken the main pillar of its rival’s business.

Most search start-ups, meanwhile, say they hope to avoid relying on advertising, if only to set themselves apart from Google.

Andi is planning to use a “freemium” business model that involves charging some of its users for a higher level of service, said Hoover. “We never want to have ads. We think ads were one of the things that corrupted Google search,” she said.

None of the companies experimenting with AI in search, however, has yet tested whether consumers will be willing to pay for something they currently receive for free. Google’s founders themselves once wrote that relying on advertising might be the wrong business model for a search engine, said Srinivas. At this stage, he added, it’s better to leave all options on the table.