AI: open-source models imperil profits of big tech’s contenders

In the 1960s, a scientist at the Massachusetts Institute of Technology created a natural language processing programme that could mimic human conversation. Named ELIZA, it was an early iteration of the chatbots running rampant across the tech sector this year. ELIZA was not a profitable endeavour. Neither are the current versions.

There are clear transformational possibilities in generative artificial intelligence. Chatbots developed using large language models (LLMs) could allow seamless communications between humans and computers.

The question for investors is whether proprietary LLMs can reliably make money for big tech. Open source LLMs could be a cheaper alternative for businesses developing bespoke applications.

LLMs have no formal definition. They are described as programmes trained on huge volumes of data available online and able to predict the next word in a sentence.

As computing power has increased the AIs have been able to carry out unsupervised learning from unstructured data. They produce some answers that surprise even their creators.

LLM complexity has leapt forward. In 2020, OpenAI released its Generative Pretrained Transformer 3, or GPT-3. This LLM had 175bn parameters.

The more parameters, the more data an LLM can process and generate. Google’s PaLM, which powers its Bard chatbot, has 540bn. OpenAI’s latest version of its LLM is GPT-4. The company has not specified the number of parameters. Pundits believe 100tn would be an accurate figure.

The processing power required for such LLMs is vast. The rule of thumb is the larger the data set used, the better the performance. This, in theory, confines LLMs to a small number of well-financed companies. 

But niche applications can function using smaller data sets. BloombergGPT, which is intended to aid analysis of information on Bloomberg data terminals, has 50bn parameters. Toronto-based start-up Cohere AI’s base model LLM has 52bn parameters.

Of more concern for companies such as Google are open source LLMs. Meta gave away its system, LLaMA, as open-source software that can be copied and used by anyone. Smaller LLMs can be built on top of it. 

Suppose enterprise users decide there is little difference between proprietary and open source LLMs when developing their own AI services, Google and OpenAI would lose their early mover advantage before they have a chance to break even.