Good robots must not be made to learn from bad human habits

Stay informed with free updates

Simply sign up to the Artificial intelligence myFT Digest — delivered directly to your inbox.

A few months ago, I found myself driving behind an autonomous (ie self-driving) vehicle in San Francisco. It was unnerving — but not because the AV was driving badly or dangerously; instead it was driving too well.

Specifically, the bot-driver dutifully halted at every “stop” sign, braked at amber lights, and stayed below the official speed limit. That forced me to do the same, somewhat to my irritation since I (like most human drivers) have hitherto sometimes gently skirted traffic rules.

It is a trivial anecdote. But it highlights an existential question hanging over the UK government’s summit on artificial intelligence next week: as we race to build systems that are getting better at the “imitation game”, what types of human behaviour should bots copy? 

Should this be our idealised vision of behaviour — say, a world where we all actually observe traffic rules? Or the land of “real” humans, where drivers creep through stop signs? And, crucially, who should decide?

The issue matters in every field where generative AI is being applied, whether finance, media, medicine or science. But the tale of AVs crystallises the issue particularly clearly, since the companies such as Waymo, General Motors and Tesla that are already running fleets in places such as Arizona and California are taking subtly different approaches. 

Consider Waymo, the AV group owned by Alphabet. Its vehicles have been roaming around Phoenix, Arizona for almost two years with an AI system that (roughly speaking) was developed using preset principles, such as the National Highway Transport Safety Administration rules.

“Unlike humans, the Waymo Driver is designed to follow applicable speed limits,” the company says, citing its recent research showing that human drivers break speeding rules half the time in San Francisco and Phoenix.

The vehicles are also trained to stop at red lights — a point that delights the NHTSA, which recently revealed that nearly 4.4m human Americans jumped red lights in 2022, and more than 11,000 people were killed between 2008 and 2021 because someone ran the lights.

Unsurprisingly, this seems to make Waymo’s cars much safer than humans. (Admittedly this is a very low bar, given that 42,000 people were killed in US car accidents last year).

But what is really interesting is that Waymo officials suspect that the presence of rule-following AVs in Phoenix is encouraging human drivers to follow rules too — either because they are stuck behind an AV or being shamed by having a bot inadvertently remind them about the traffic rules. Peer pressure works — even with robots.

There is little research on that — yet. But it reflects my own experience in San Francisco. And a (limited) study by MIT shows that the presence of AVs on a road can potentially improve the behaviour of all drivers. Hooray.

However Elon Musk’s Tesla has taken a different tack. As Walter Isaacson’s biography of Musk notes, initially Musk tried to develop AI with preset rules. But then he embraced newer forms of generative or statistical AI (the approach used in ChatGPT). This “trains” AI systems how to drive not with preset code but by observing real human drivers; apparently 10m video clips from existing Tesla cars were used.

Daval Shroff, a Tesla official, told Isaacson that the only videos used in this training were “from humans when they handled a situation well”. This means that Tesla employees were told to grade those 10m clips and only submit “good” driving examples for bot training — to train bots in good, not bad, behaviour.

Maybe so. But there are reports that Tesla AVs are increasingly mimicking humans by, say, creeping across stop signs or traffic lights. Indeed, when Elon Musk live-streamed a journey he took in August in an AV, he had to intervene manually to stop it jumping a red light. The NHTSA is investigating.

Of course, Musk might retort that all drivers occasionally need to break rules in unusual circumstances to preserve safety. He might also retort that it is natural for companies to take different approaches to AI, and then let customers choose; that is how corporate competition normally works.

However, some regulators are worried: although GM’s Cruise produced data earlier this year showing a good overall safety record, this week California’s Department of Motor Vehicles demanded that the company stop operating unmanned AVs after an accident. And the rub is that while regulators can theoretically scrutinise AVs using preset rules (if outsiders have access to the code), it is harder to monitor generative AI, since the consequence of mimicking “real” human behaviour is so unpredictable — even for their creators.  

Either way, the key point investors need to understand is that variants of this problem will soon haunt fields such as finance too, as Gary Gensler, the Securities and Exchange Commissioner recently told the FT. Should AI-enabled players in financial markets be programmed with preset, top-down rules? Or learn by mimicking the behaviour of humans who might “arbitrage” (ie bend) rules for profit? Who decides — and who has liability if it goes wrong? 

There are no easy answers. But the more embedded AI becomes, the harder the challenge, as attendees at next week’s summit know. Maybe they should start by pondering a traffic light.

gillian.tett@ft.com