Meta finds AI-generated content on Israel-Gaza used ‘deceptively’ on Facebook, Instagram; takes action

Meta had found “likely AI-generated” content used deceptively on its Facebook and Instagram platforms, including comments on the Israel-Gaza conflict published below posts from global news organisations and US lawmakers, the social media company said in its quarterly security report. 

The social media accounts posted as Jewish students, African Americans, and other concerned citizens are targeting the US and Canadian audiences. The campaign was attributed to Tel Aviv-based political marketing firm STOIC, Reuters reported. 

While Meta has encountered AI-generated basic profile photos in influence operations since 2019, this report is the first to reveal the use of text-based generative AI technology, which has existed since late 2022.

Researchers warned that generative AI could lead to more effective disinformation campaigns and sway elections as it can quickly and cheaply produce human-like text, images, and audio. 

Meta security executives reported that they swiftly dismantled the Israeli campaign and did not believe that advanced AI technologies hindered their capacity to disrupt influence networks, which are organized efforts to propagate specific messages.

They added that the executives had not seen such networks deploying AI-generated imagery of politicians realistic enough to be confused for authentic photos. 

Meta head of threat investigations Mike Dvilyanski said, “There are several examples across these networks of how they use likely generative AI tooling to create content.”

“Perhaps it gives them the ability to do that quicker or to do that with more volume. But it hasn't really impacted our ability to detect them,” Mike said as quoted by Reuters. 

Besides the STOIC network, Meta also shut down an Iran-based network focused on the Israel-Hamas conflict. However, it did not identify any use of generative AI in that campaign.

It is worth mentioning that Meta and other tech giants have grappled with how to address the potential misuse of AI technologies, especially in elections. 

Researchers have discovered instances where image generators from companies like OpenAI and Microsoft have created photos containing voting-related disinformation, even though these companies have policies prohibiting such content.

The companies have highlighted using digital labelling systems to identify AI-generated content at the time of creation. However, these tools do not apply to text, and researchers question their effectiveness.

(With Reuters inputs)