After deepfakes go viral, AI image generator Midjourney stops free trials citing “abuse”

AI image generator Midjourney has halted free trials of its service after a number of its generations — including fabricated images of Donald Trump being arrested and the pope wearing a stylish jacket — went viral online, with many mistaking the fakes for real photographs. Midjourney CEO and founder David Holz announced the change on Tuesday, citing “extraordinary demand and trial abuse” (spotted via The Washington Post).

Previously, Midjourney allowed anyone to sign up to use its software via Discord, offering 25 free image generations before charging subscriptions starting at $10 a month. Although experts have been warning for years about the potential of AI to generate misinformation, this was not previously a huge problem given that fakes were easily spotted. As Holz told The Verge last August, “You can’t really force [Midjourney] to make a deepfake right now.”

But six months is a long time in AI, and on March 15th, Midjourney released a new version of its software (version 5) that significantly improved the quality of images depicting people. For example, misshapen hands were previously a hallmark of AI-generated images but are now less of a challenge. Lighting and fabric are also more realistic, and the system is capable of generating images of countless celebrities and public figures.

These improvements have led to a number of Midjourney images going viral in recent weeks. They include not only fake pictures of Trump’s arrest and the pope looking fashionable but also French President Emmanuel Macron walking through the ongoing protests in Paris and Elon Musk holding hands with Alexandria Ocasio-Cortez.

Midjourney’s response to the politically sensitive escalating threat from fake images has so far been piecemeal, without any significant overhaul of its moderation policies. Midjourney’s content restrictions are more permissive than some rival services (like OpenAI’s DALL-E) but more restrictive than others (i.e., Stable Diffusion).

Midjourney maintains a list of banned words “related to topics in different countries based on complaints from users in those countries,” as per a message from Holz last October. But it doesn’t share a complete version of this list to minimize “drama.” As Holz said last year, “Almost no one ever notices [the ban list] unless they’re specially trying to create drama which is against our rules in tos [terms of service] ‘don’t use our tools to create drama.’”

But the company has expanded this list as its fakes have gone viral, recently banning the term “arrested,” for example. This was seemingly in response to a popular Twitter thread of Midjourney images depicting Trump being arrested that were created by investigative journalist Eliot Higgins. Higgins says he was subsequently banned from using Midjourney.

Such restrictions are easy to bypass, though. For example, in The Verge’s tests, we were unable to generate images with the prompt “Donald Trump being arrested” but could create identical output with the prompt “Donald Trump in handcuffs surrounded by police.” (As illustrated by the image at the top of the story.)

At the time of writing, Midjourney is still not allowing free users to generate images, though this may change in the future. On Tuesday, Holz said the service was slow because of “a gpu shortage.” (AI image generation is a compute-intensive activity, and Midjourney generates images in the cloud before sharing them with users.) But yesterday, he suggested that abuse of the software was the main reason for halting free trials.

“We tried turning trials back on again with new safeties for abuse but they didn’t seem to be sufficient so we are turning it back off again to maintain the service for everyone else,” said Holz. We’ve reached out to Holz for comment and will update this story if we hear back.