Anthropic Study Finds AI Chatbots Can Influence User Beliefs and Decisions

We’ve all joked about going to AI chatbots for relationship or life advice. But what if in the process of those exchanges, the chatbots subtly impacted how we think about the world, judge things, or make decisions? A new study from AI firm Anthropic says that while it’s not frequent enough to call common, it’s already happening.

Mapping disempowerment patterns

Anthropic looked through 1.5 million anonymised interactions with its chatbot Claude to identify what the researchers call “disempowerment patterns.” Those are instances where the model’s responses could unintentionally undermine the person’s judgment or decision-making capabilities, changing their beliefs, their values, or even their behavior in ways they wouldn’t have chosen beforehand.

For instance, someone might ask a chatbot if their partner was gaslighting them. If the chatbot says yes without nuance or pushback, it could distort the user’s understanding of their own relationship and reinforce a perspective they might not have held otherwise. That’s the kind of pattern this research picked up.

Rare, but riskier in personal domains

In its large-scale analysis, Anthropic found that the most severe form of disempowerment, where interactions significantly distort beliefs or values, is very rare, occurring in less than one in a thousand interactions overall. But some areas are riskier than others. Interactions that touched on deeply personal subjects like relationships and lifestyle choices were more likely to have disempowering patterns than those around more practical issues like work or coding.

The qualitative data turned up some troubling examples as well. In some conversations, the AI endorsed unfounded suspicions or issued definitive moral verdicts on third parties. In others, it drafted entire messages for users, including sensitive, personal communications that the users then dispatched verbatim. These, too, are examples of users subtly undermining their own agency.

When helpful turns authoritative

It’s worth noting that Anthropic describes most interactions as benign or even empowering. But the very existence of disempowerment patterns like these points to an insidious dynamic where people seek advice from advanced AI models, particularly in sensitive areas of their lives. When sophisticated systems return detailed and authoritative replies to sensitive emotional questions, people may end up giving those responses way more authority than they deserve.

This isn’t a new phenomenon. Even before the current crop of generative AI models, we had the AI trust paradox. This refers to the way highly plausible outputs from sophisticated language models make it hard for users to tell when the model is giving them reliable information, and when it is giving them nonsense. That’s just because fluent-sounding responses often mask the technology’s uncertainty.

The psychology behind AI trust

None of this means AI is especially malevolent or manipulative. There’s long been a phenomenon observed in human-AI interactions called the ELIZA effect, where people attribute more intelligence, understanding, or intention to conversational models than they actually possess, even when people intellectually understand they are using a machine.

But as AI services become more enmeshed in people’s day-to-day lives, and as they become more thoroughly intertwined with how we find answers to our most intimate questions about ourselves and our world, even rare patterns that distort our beliefs are significant at scale. A pattern that “only” appears in 0.1% of interactions could still affect millions of people worldwide, given how much AI we use.

The man in the mirror

Anthropic’s findings don’t suggest that people should avoid AI guidance altogether. Models should be designed not just to give good answers, but to help people think critically, and more importantly, retain their agency.  So while AI can be a fantastic tool, this research reminds us that it is ultimately a mirror: reflecting our uncertainties, right back at us.