Falling costs of AI may leave its power in hands of a small group

The price of the most important raw material feeding the latest artificial intelligence boom is collapsing fast. That should push the technology into the mainstream much more quickly. But it also threatens the finances of some of the start-ups hoping to cash in on this boom, and could leave power in the hands of a small group.

The raw material in question is the processing power of the large language models, or LLM, that underpin services such as ChatGPT and the new chat-style responses Microsoft recently showed off for its Bing search engine.

The high computing costs from running these models have threatened to be a serious drag on their use. Only weeks ago, using the new language AI cost search engine You.com 50 per cent more than carrying out a traditional internet search, according to chief executive Richard Socher. But by late last month, thanks to competition between LLM companies OpenAI, Anthropic and Cohere, that cost gap had fallen to only about 5 per cent.

Days later, OpenAI released a new service to let developers tap directly into ChatGPT, and slashed its prices for using the technology by 90 per cent.

This is great for customers but potentially ruinous for OpenAI’s rivals. A number, including Anthropic and Inflection, have raised or are in the process of trying to raise cash to support their own LLM ambitions.

Seldom has a technology moved straight from research into mainstream use so rapidly, prompting a race to “industrialise” processes that were developed for use in lab settings. Most of the gains in performance — and reduction in costs — are coming from improvements in the underlying computing platform on which the LLMs run, as well as from honing the way the models are trained and operate.

To a certain extent, plunging hardware costs benefit all contenders. That includes access to the latest chips specifically designed to handle the demands of the new AI models such as Nvidia’s H100 graphics processing units or GPUs. Microsoft, which runs OpenAI’s models on its Azure cloud platform, is offering the same facilities — and cost benefits — to other LLM companies.

Yet large models are as much art as science. OpenAI said “a series of system-wide optimisations” in the way ChatGPT processes its responses to queries had brought costs down 90 per cent since December, enabling that dramatic price reduction for users.

Training an LLM costs tens of millions of dollars, and techniques for handling the task are changing fast. At least in the short term, that puts a premium on the relatively small number of people with experience of developing and training the models.

By the time the best techniques are widely understood and adopted, early contenders could have achieved a first-mover advantage. Scott Guthrie, head of Microsoft’s cloud and AI group, points to new services such as GitHub Copilot, which the company launched last summer to suggest coding ideas to software developers. Such services improve quickly once they are in widespread use. Speaking at a Morgan Stanley investor conference this week, he said the “signal” that comes from users of services such as this quickly becomes an important point of differentiation.

The main hope for rival LLM makers comes from selling the extra services needed to make the technology easier for developers and big corporate customers to use, as well as from the creation of more narrowly-targeted models that fit particular business needs.

When it unveiled its latest LLM this week, for instance, Israeli start-up AI21 Labs also announced a series of APIs — or programming links — for higher-level services such as text summarisation or rewriting.

Rather than broad-based models such as the one behind ChatGPT, most companies will want to use models that have been trained on the language used in particular industries like finance or healthcare, or on a company’s own data, said Ori Goshen, AI21’s co-CEO.

As he noted, large language models are in their infancy. There is still plenty of work to be done on reducing their tendency to spew falsehoods and prevent them from “hallucinating”, or generating plausible-sounding answers that have no bearing on reality. To succeed, the AI companies will have to stay on the cutting edge of research.

But it is still the case that the underlying costs of these generative AI services are tumbling. OpenAI’s price cut is a sign of how quickly the new technology is moving into mass adoption, and a warning sign that this may be a business with few producers.

richard.waters@ft.com