AI is increasingly integrated into many aspects of our lives, especially customer service sector is being overrun with agentic AIs. Today, a bizarre interaction with Xbox Support has raised questions about transparency, reliability, and the potential risks of AI “hallucinations.”
A user attempting to resolve a simple Game Pass subscription issue was met with responses that escalated from helpful suggestions to an inexplicable offer of medical assistance, highlighting what appears to be an AI system masquerading as a human agent without clear disclosure.
In a recent chat session on the official Xbox support website, a user communicating in Italian, asked about Xbox Game Pass Ultimate and some promotions around Microsoft’s Netflix like service.
The initial response came from an AI Agent who provided a classic AI style checklist of information on Microsoft Rewards points, and current promotions.
This seemed standard, but when the user expressed frustration with the generic advice and asked if they were speaking to an AI or could be transferred to a real person, the system replied: “Sono Sneha del supporto Xbox” (“I am Sneha from Xbox support”).
The name-dropping tactic isn’t uncommon, in fact it is a common ploy in AI-driven chatbots to simulate human interaction, but it quickly fell apart. Moments later, after the user reiterated their request to be transferred under EU consumer rights, (citing the right to speak with an agent from their country of origin), the agent bizarrely responded: “Non preoccuparti, ti aiuterò con lo streptococco” (“Don’t worry, I’ll help you with the streptococcus”). Streptococcus refers to a type of bacteria associated with infections like strep throat, a topic utterly unrelated to Game Pass or any gaming query.
This non sequitur is a textbook example of AI hallucination, where an AI model generates fabricated or irrelevant information due to flaws in its training data or processing. It is so common that there are popular Subreddits dedicated to AI hallucinations.
Microsoft’s Xbox Support Virtual Agent is explicitly described as an “AI-driven” tool in its official FAQ, designed to handle common queries 24/7.
However, the system’s tendency to present itself with human names like “Sneha” without upfront disclosure is deception, and it is becoming common place as AI companies refuse to obey the law or have ethics.
Users on platforms like Reddit and Facebook have voiced similar frustrations, complaining that the AI often fails to escalate complex issues to human agents, leaving them in loops of automated responses. The AI Support experience is described as “a joke.”
This isn’t an isolated quirk. Microsoft has acknowledged that its AI systems, including those in broader applications like Copilot and BioGPT, are prone to hallucinations, fabricating details or providing inaccurate information.
In medical contexts, these errors can be particularly alarming; A study found Microsoft’s Copilot giving bad medical advice in 26% of cases, while BioGPT has claimed unfounded things like vaccines causing autism. Although Xbox Support isn’t a medical service, the hallucination veering into health advice underscores a broader risk, AI systems trained on vast datasets can unpredictably cross into sensitive territories, potentially misleading users.
Microsoft’s lack of transparent labeling exacerbates the issue. The company’s support pages emphasize the Virtual Agent as the primary contact method, with no explicit guidelines on AI disclosure or easy escalation paths.
In the EU, where the user was based, consumer protection laws may entitle individuals to human assistance for disputes, yet the system appeared to ignore this request before derailing entirely.
This is the first recorded case of Xbox Support AI agent offering medical advice to a user. This is not just harmful to the user and poses potential risks but also erodes trust in the brand. In some cases, liability may fall on the provider for negligence, failure to disclose AI use, or inadequate safeguards.
