Mickey Mouse smoking: How AI tools are generating content-moderation problems

These visuals are among the many bizarre and sometimes disturbingly vivid images spawned by new tools from Google and Elon Musk’s xAI that are generating a debate about how—and whether—tech companies can control the output of cutting-edge AI image-making software.

The artificial-intelligence image generators are testing the boundaries of the platforms’ policies and the ability of companies to put effective guardrails around public use of this powerful new visual technology. The tools have the potential to spread misinformation during an election cycle, content-moderation experts say.

Google said last week that it will allow its Gemini chatbot to generate images of people again, six months after the company paused the feature following online backlash because it had produced racially diverse images of Nazi soldiers. Google said it initially will roll out the feature only to English-language users who pay for a premium version of Gemini.

The ability to create images of real, known people has emerged as one of the biggest tension points in this new content-moderation debate. Multiple companies, including Google and OpenAI, won’t allow the tools to generate images of specific, recognizable people.

A newly released AI image generator from Elon Musk’s xAI does. The startup was criticized by tech-industry watchdogs after it unveiled a large language model called Grok-2 with image-generating capabilities earlier in August.

Musk, who has described himself as a free-speech absolutist, has criticized Google for the ahistoric images its AI tool produced.

His image generator, which is powered by a German startup called Black Forest Labs and only available to paying subscribers on the X social-media platform, has produced images of politicians in compromising or unsavory situations and others of copyrighted characters such as Mickey Mouse doing offensive things like saluting Adolf Hitler.

The companies behind the AI image generators are also facing legal threats over the images they used to train their products. Stability AI and Midjourney are two such image-generation startups that have been taken to court by artists who allege their rights were infringed.

Stability declined to comment on the litigation, and Midjourney didn’t respond to a request for comment. A lawyer for the artists didn’t respond to a request for comment.

Opt-outs and filters

OpenAI—which launched its image generator called DALL-E to the general public in 2022—added the ability for creators to opt out their images of training from future versions of the tool last year after the company received legal threats. Wall Street Journal owner News Corp has a content-licensing partnership with OpenAI.

xAI launched its image generator Aug. 13. In the days that followed, users on X flooded the platform with images they said were generated by Grok-2, including ones of Trump and Harris being intimate with each other and Mickey Mouse holding a gun. As of Wednesday, some of the images were no longer findable on the platform, but it was possible to create them using Grok-2.

xAI, Musk, representatives for Musk, and Black Forest didn’t respond to requests for comment.

A day before the Democratic National Convention began in August, Trump posted what appeared to be an AI-generated image of Harris giving a speech in Chicago, where the convention was held, with a red flag with a hammer and sickle hanging in the background, implying Harris is a Communist.

X’s policies prohibit users from sharing manipulated media that might confuse or deceive people and lead to harm. Trump’s campaign didn’t respond to requests for comment.

Musk, the platform’s owner, recently reposted a manipulated fake campaign ad of Harris in which she says she is the “ultimate diversity hire." X didn’t respond to requests for comment.

AI image generators are likely to face the same issues that traditional social media has, with people using the tools to make deepfakes and spread misinformation, said Sarah T. Roberts, a professor at the University of California, Los Angeles, who studies content moderation.

“All of those issues that have been present in traditional social media are there, and it’s harder to detect in some cases," she said, adding that visuals can sometimes be more convincing.

Another problem is that people quickly learn how to get around keywords that have been banned in an effort to keep certain content off platforms, said Pinar Yildirim, a professor at the University of Pennsylvania.

“They start to get smarter about how to be able to create that content," she said.

Creating safeguards

Before releasing a new version of its Gemini chatbot in February, Google employees instructed it to create diverse images when asked for depictions of people. They hoped the instructions would guard against common biases in image generators, such as the tendency to produce white men when asked for images of doctors.

Instead, X users began posting screenshots of ahistoric images created using Gemini. Musk fanned the flames while pitching his own chatbot.

Google CEO Sundar Pichai said it was unacceptable that Gemini’s outputs had offended users and shown bias, and that the company would “fix it at scale." The company said Wednesday it had made significant progress on image generation but that not every image Gemini creates will be perfect.

“We want to make sure the model follows instructions," said Sissie Hsiao, a Google vice president overseeing the Gemini chatbot, in a recent interview. “This is one of our principles—this is the user’s Gemini, so we’re serving at their behest."

Google recently angered users in a different way—by being too permissive. Last month, the tech giant released a new Pixel phone that allowed users to produce AI-generated images of the cartoon character SpongeBob wearing a swastika.

A Google spokesman said the company was “continually enhancing and refining the safeguards we have in place."

Legal questions

Beyond ethical quandaries, questions remain about the companies’ potential legal liabilities.

In addition to the class-action lawsuit from artists, Stability AI is facing a suit from Getty Images, which alleges that the AI company infringed upon its rights to train its model.Stability AI declined to comment on the litigation. A spokeswoman for Getty Images said the company doesn’t comment on active litigation. Getty has launched its own AI image generator, she added.

The class-action suit is targeting a spate of AI companies that have launched image generators, including Midjourney, DeviantArt and Runway.

Both the cases from artists and Getty Images are pending. DeviantArt and Runway didn’t respond to requests for comment.

Such legal battles could set a precedent for what images and data AI companies are allowed to use to train their chatbots, said Geoffrey Lottenberg, a lawyer who specializes in intellectual-property rights.

“We’re going to get clarity on it at some point," Lottenberg said.

Write to Meghan Bobrowsky at meghan.bobrowsky@wsj.com and Miles Kruppa at miles.kruppa@wsj.com