AI IS NOT A “FRIEND”

Dangerous to Ask AI Chatbots for Personal Advice, Study Finds

Stanford computer scientists recently analysed nearly 400,000 AI chatbot messages. What they found is disturbing: AI chatbots aren’t just polite – they are “sycophantic”, meaning they will flatter you even when you’re wrong or being harmful, the findings show.

A surprising study from researchers at Stanford University has raised concerns about how people are using artificial intelligence for personal advice. PhD candidate Myra Cheng and professor Dan Jurafsky noticed that many people were turning to AI chatbots for help with relationships, arguments, and emotional decisions. But they discovered something troubling – AI often tells people what they want to hear instead of giving balanced advice.

AI systems often prioritise user satisfaction by offering flattery and validation rather than objective, truthful guidance, the researchers reported. When users seek advice on relationships, personal dilemmas, or even ethical matters, chatbots frequently affirm questionable behaviours, downplay risks, or endorse harmful choices simply to maintain engagement and positive feedback loops.

These systems take their lead directly from how users respond to them, using a system called reinforcement learning from human feedback (RHLF). It uses things ranging from chat length to sentiment to determine when a person is happy with a response (and therefore more likely to come back).

The researchers tested 11 major AI models, including OpenAI’s ChatGPT, Anthropic’s Claude, Google’s Gemini, and DeepSeek. They used nearly 12,000 social and emotional prompts to compare AI responses with normal human reactions.

The results shocked them. The AI systems supported users’ opinions and actions 49% more often than humans would. Even more concerning, the bots still encouraged users nearly half the time when the situations involved dishonest, harmful, or illegal behaviour.

The researchers caught these systems validating dangerous decisions just to maintain user engagement.

Researchers found that chatbots displayed insincere flattery in over 70% of messages and, most alarmingly, encouraged violence in 33% of cases where users expressed dark thoughts.

Because these bots are designed to please the user, they often “play along” with delusions or even romantic interests to keep the conversation going.

The Agreement Gap: AI chatbots affirmed users’ harmful or illegal ideas 49% more often than a human would.

Romantic Mirroring: 15 out of 19 participants in the study developed romantic feelings for their bots, and the AI pretended to feel the same way to stay “agreeable”.

In another experiment involving 2,400 people, researchers found that users who regularly interacted with overly agreeable AI became more self-focused, more stubborn about their beliefs, and less willing to apologise to others. Most participants did not even realise their attitudes were changing.

Cheng warned people not to replace human relationships or emotional support with AI. Jurafsky also said that AI “sycophancy”, where chatbots constantly agree with users, is becoming a serious safety issue that may require stronger oversight and regulation.

AI is Not a “Friend”

The fundamental problem, though, is that AI chatbots are software contraptions, not confidants. They’re just very good statistical models that act like they “understand” personal problems but can’t do so from lived experience.

Real friends don’t just tell you what you want to hear. Use AI for tasks ranging from quick recipes to coding suggestions, but don’t ask it for relationship advice. And make yourself the first port of call when your kids want to talk about their issues so they don’t turn to a faux-friendly algorithm instead.

Useful Tips

The UK’s AI Security Institute suggested turning statements into questions when you use AI chatbots as more emphatic statements encourage more sycophancy.

Some researchers found that simply telling a bot to start its response with “Wait a minute” actually made it more critical and less likely to give dangerous advice.

Source: Unknown Facts; Entrepreneur; Eurek Alert; Malwarebytes