AI Chatbots’ Hidden Bias: How Chatbots Can Influence Opinions | Futurity

The way we gather information is changing, and a new study from Yale University suggests that even a simple question posed to an AI chatbot can subtly shift your perspective. As people increasingly turn to these tools for quick answers, researchers have found that the underlying biases within the chatbot’s programming can influence users’ social and political opinions, even when the chatbot isn’t explicitly trying to persuade them.

This isn’t about chatbots actively campaigning or presenting one-sided arguments. The study, published in PNAS Nexus, reveals that the way information is framed – the subtle nuances in the narratives generated by these AI systems – can nudge our thinking in predictable directions. The findings build on prior research demonstrating that AI-generated content can be persuasive when designed to be, but this study highlights a more insidious effect: influence happening unintentionally.

How Chatbots Shape Understanding of History

The Yale team, led by assistant professor of sociology Daniel Karell, focused on how chatbots present historical events. They examined two case studies: the Seattle General Strike of 1919 and the Third World Liberation Front protests at the University of California, Berkeley, in 1968. These events were chosen for their complex social and political contexts, offering ample opportunity for subtle framing effects.

Researchers presented 1,912 participants with summaries of these events, sourced either from GPT-4o, an OpenAI chatbot released in 2024, or from Wikipedia. A separate group read summaries deliberately framed with either liberal or conservative viewpoints. The results were telling. Both the default AI summaries and those with a liberal slant led participants to express more liberal opinions about the events compared to those who read the Wikipedia entries. Conversely, summaries framed with a conservative perspective prompted more conservative responses.

It’s important to note that the shifts in opinion weren’t dramatic. Karell describes the effects as “modest,” moving individuals from a moderate stance to a somewhat liberal one, for example. However, he cautions that these small influences could accumulate over time with frequent chatbot use. The study’s lead author, Matthew Shu, a 2025 Yale College graduate, emphasized the potential for compounding effects.

Latent Bias: The Hidden Influence

The key takeaway is the concept of “latent bias.” Large language models (LLMs), the technology powering these chatbots, are trained on massive datasets of text and code. These datasets inevitably reflect the biases present in the real world – and, crucially, the biases of those who created the data. These biases aren’t explicitly programmed into the chatbot; they emerge as subtle patterns in the way the LLM processes and generates information.

As the Yale researchers explain, these latent biases subtly influence the framing of narratives. The chatbot isn’t intentionally trying to persuade you, but the way it presents information – the words it chooses, the details it emphasizes – can subtly steer your interpretation. This represents different from a deliberately persuasive argument, making the influence harder to detect and resist.

Political Ideology and Susceptibility

The study likewise explored whether existing political beliefs moderate the impact of chatbot framing. Participants were asked to self-identify their political ideology. The researchers found that liberal framing in AI summaries consistently led to more liberal opinions across all ideological groups. However, conservative framing only had a statistically significant effect on those who already identified as politically conservative.

This suggests that conservative framing in GPT-4o’s output is more likely the result of deliberate prompting bias, although liberal framing may stem from a combination of latent and prompted biases. This distinction is important for understanding the source of the influence and potentially mitigating it.

Beyond the Study: Broader Implications

The findings from Yale University align with growing concerns about the potential for AI to influence public opinion. A December 2025 report from Nature highlighted that AI chatbots can sway voters in major elections, potentially having a greater impact than traditional campaigning. The opaque nature of AI development, as Karell points out, is particularly concerning. Unlike Wikipedia, where editing processes are transparent and open to scrutiny, the inner workings of LLMs are largely hidden from public view. This lack of transparency raises questions about accountability and the potential for manipulation.

Further research, including a systematic review published in PMC, demonstrates the growing role of AI-powered chatbots in various fields, including healthcare. While these tools offer potential benefits, understanding their inherent biases is crucial for responsible implementation.

What to Consider When Using Chatbots for Information

This study doesn’t suggest that chatbots are inherently untrustworthy or that we should avoid using them altogether. However, it does underscore the importance of critical thinking and media literacy. When relying on chatbots for information, especially about complex or controversial topics, it’s essential to be aware of the potential for subtle bias.

Consider cross-referencing information with multiple sources, including reputable news organizations, academic research, and official government websites. Be mindful of the framing of the information presented and ask yourself whether the chatbot might be subtly influencing your interpretation. Remember that chatbots are tools, and like any tool, they can be used – and misused – in ways that shape our understanding of the world.

Looking Ahead: Addressing Bias in AI

The researchers emphasize that addressing bias in LLMs is a complex challenge. It requires careful curation of training data, development of techniques to detect and mitigate bias, and greater transparency in the development process. Ongoing research is focused on these areas, and it’s likely that we’ll see further advancements in the coming years. For now, a healthy dose of skepticism and a commitment to seeking diverse perspectives are essential when navigating the increasingly AI-driven information landscape.