Emotional ties to AI? OpenAI warns of new risks with ChatGPT Voice Mode

Sam Altman's OpenAI has raised concerns about the potential emotional attachment users may develop with its recently launched Voice Mode feature for ChatGPT. This warning was detailed in the company's "System Card" for GPT-4o, a comprehensive document that examines the potential risks and safeguards associated with the AI model. Among the various risks identified, the possibility of users anthropomorphizing the chatbot—attributing human-like characteristics to it—has emerged as a significant concern.

The Voice Mode, which allows ChatGPT to mimic human speech and convey emotions, may lead to users forming social connections with the AI, OpenAI cautioned. This concern is not purely theoretical; during early testing phases, including red-teaming (a process where ethical hackers simulate attacks to identify vulnerabilities) and internal trials, the company observed instances of users developing emotional bonds with the AI.

In one notable case, a user expressed a sense of loss, saying, "This is our last day together," indicating a level of attachment that prompted OpenAI to take note.

OpenAI is particularly worried about the broader societal impacts of such attachments. The company highlighted the potential for AI-human interactions to alter social norms. For instance, in human conversations, interrupting someone mid-sentence is generally considered rude, yet users can freely interrupt ChatGPT without repercussions. This could potentially normalize behaviors that are considered impolite in human-to-human interactions.

Moreover, the company warned that as users grow more accustomed to socializing with AI, it could negatively impact their relationships with other people. While the technology might offer companionship to those who are lonely, it could also undermine the quality of healthy, human connections.

Currently, OpenAI does not have a definitive solution to this issue but plans to continue monitoring the situation. The company expressed its commitment to further research the implications of emotional reliance on AI and how its deeper integration into daily life could influence user behavior.