AI Chatbots & Your Health: Risks of Misdiagnosis & When to Use Them

The rapid integration of artificial intelligence into healthcare is offering new avenues for patients to access information and support, but emerging research underscores a critical caution: these tools are not always reliable, particularly when it comes to medical advice. With platforms like OpenAI’s ChatGPT gaining traction – the company reports over 40 million daily health-related queries according to a recent announcement – understanding the limitations of AI in medical contexts is paramount.

The Challenge of Effective AI Consultation

A recent study published in Nature Medicine highlights a significant gap between the potential of AI chatbots and their actual utility for patients. Researchers simulated real-world scenarios, asking participants to consult AI tools with hypothetical medical cases. The results were sobering: participants correctly identified the condition in only about one-third of instances. Even more concerning, only 43% made the correct decision regarding next steps, such as whether to seek emergency care or manage the condition at home. The study, led by researchers at Oxford University, points to a fundamental issue: many individuals lack the expertise to effectively query these systems.

Andrew Bean, an AI systems researcher at Oxford University and co-author of the study, explains that the quality of the response is heavily dependent on the phrasing of the question. “Doctors are trained to ask questions about symptoms you might not have realized you should have mentioned,” he says. A subtle difference in description can lead to drastically different recommendations. For example, one participant describing “the worst headache I’ve ever had” was advised to go to the emergency room, while another, using less emphatic language, was told to take aspirin and rest – despite both presenting the same underlying condition.

Diagnostic Accuracy: A Mixed Picture

While AI can sometimes struggle with nuanced symptom interpretation, it’s not without its strengths. Some studies have shown that large language models can match, or even outperform, physicians in specific diagnostic reasoning tasks. Research published in the New England Journal of Medicine demonstrates this potential in certain scenarios. However, Bean emphasizes that these successes occur in controlled, clinical settings, far removed from the “messy” reality of how people actually interact with AI chatbots.

Under-Triage and the Urgency Gap

Beyond diagnostic accuracy, another study revealed a concerning tendency for AI to underestimate the severity of medical emergencies. Researchers presented AI bots with various medical scenarios and found that in 52% of emergency cases, the bots “under-triaged” the condition, suggesting a less urgent course of action than warranted. In one particularly alarming example, the AI failed to recommend emergency care for a hypothetical patient experiencing diabetic ketoacidosis with impending respiratory failure – a potentially life-threatening situation. This research, led by Dr. Girish Nadkarni at Mount Sinai, suggests that AI may struggle to accurately assess the time-sensitive nature of certain medical conditions.

OpenAI acknowledges these concerns, stating that the initial study utilized an older version of ChatGPT that has since been updated to address some of the identified issues. However, the company also cautions that the study’s methodology may not accurately reflect how people typically use the platform.

The Value of AI as a Complement, Not a Replacement

Despite the inherent risks, many healthcare professionals believe AI can still play a valuable role in patient care. Dr. Robert Wachter, a physician at UC San Francisco and author of A Giant Leap: How AI Is Transforming Health Care and What That Means for Our Future, encourages patients to utilize these tools, particularly given the challenges of accessing affordable and timely medical care. “The advice you secure from the tools is substantially better than nothing and better than what you would get from your second cousin,” he argues.

However, Wachter is emphatic that AI should not be viewed as a substitute for a qualified physician. Adam Rodman, a hospitalist researching AI programs at Harvard Medical School, suggests a strategic approach: use AI to become better informed *before* and *after* consulting with a doctor. He envisions AI as a tool to facilitate more efficient and productive conversations with healthcare providers, allowing patients to actively participate in their care decisions.

“A excellent time to use a large language model is when you’re about to go see a doctor — or after you see your doctor,” says Rodman. “You’ll see no downsides to better understanding your health.”

The Evolving Landscape of AI in Healthcare

The integration of AI into healthcare is an ongoing process and experts anticipate that both AI systems and human practitioners will continue to refine their interactions. Rodman hopes to see AI used in a way that enhances the human element of medicine, fostering communication and streamlining bureaucratic processes. However, he also expresses concern about the potential for AI to erode trust in the doctor-patient relationship, particularly if sensitive diagnoses are delivered by a bot rather than a human. Studies have shown that treating healthcare as a marketplace product can diminish patient trust in their physicians.

The future of AI in healthcare hinges on a careful balance between leveraging its potential benefits and mitigating its inherent risks. As these technologies continue to evolve, a critical focus on accuracy, transparency, and the preservation of the human connection will be essential to ensure that AI serves as a true ally in promoting health and well-being.

Navigating the Future: A Collaborative Approach

The consensus among healthcare professionals is clear: AI is not a replacement for human expertise, but a tool that can augment and enhance the patient-physician relationship. The key lies in responsible implementation, ongoing evaluation, and a commitment to prioritizing patient safety and well-being. As AI continues to evolve, a collaborative approach – involving researchers, clinicians, and patients – will be crucial to unlocking its full potential while safeguarding against its potential pitfalls.