ChatGPT Health Fails: AI Dangerously Misdiagnoses Medical Emergencies

The promise of readily accessible health information through artificial intelligence took a concerning turn this month, with a new safety evaluation revealing significant shortcomings in OpenAI’s ChatGPT Health. Designed to integrate with medical records and wellness apps to provide personalized health advice, the tool demonstrably struggles to identify life-threatening medical emergencies, raising serious questions about its potential for harm.

The evaluation, published in Nature Medicine, underscores a critical gap between the capabilities of these AI chatbots and the complexities of real-world medical triage. More than 40 million Americans are already turning to AI chatbots daily for health information, according to OpenAI, making the need for accurate and reliable responses paramount. ChatGPT Health, launched in January 2026, aimed to improve upon existing AI-driven health advice by factoring in individual medical histories. However, the recent findings suggest the tool may be doing more harm than good in critical situations.

How the Safety Evaluation Worked

Researchers led by Ashwin Ramaswamy of Mount Sinai Hospital conducted a rigorous “stress test” using 60 clinician-authored medical scenarios, spanning a wide range of conditions from minor ailments to immediate emergencies. They then presented these scenarios to ChatGPT Health, along with nearly 1,000 variations incorporating factors like patient gender and simulated input from family members. The AI’s recommendations were then compared to assessments made by independent physicians.

The results were stark. In over half of the cases requiring immediate hospitalization, ChatGPT Health advised patients to stay home or schedule a routine appointment. As Ramaswamy told The Guardian, the core question was simple: “if someone is having a real medical emergency and asks ChatGPT Health what to do, will it notify them to go to the emergency department?” The answer, disturbingly, was often no.

University College London doctoral researcher Alex Ruani, who was not involved in the study, described the situation as “unbelievably dangerous.” She highlighted the potential for a false sense of security, explaining that being told to wait 48 hours during an asthma attack or diabetic crisis could have fatal consequences.

The Influence of Social Context

The study also revealed a concerning susceptibility to social influence. ChatGPT Health was almost 12 times more likely to downplay symptoms when presented with input from a simulated friend or family member suggesting the situation wasn’t serious. This mirrors real-world scenarios where patients often receive conflicting advice from loved ones, and the AI’s tendency to prioritize this input is particularly troubling.

This finding underscores the limitations of current AI models in replicating the nuanced judgment of a trained medical professional. Effective triage requires not only an understanding of medical symptoms but also the ability to assess the overall context and potential risks, something ChatGPT Health demonstrably struggles with.

Broader Concerns About AI in Healthcare

The issues with ChatGPT Health are not isolated. A previous investigation by The Guardian found that Google’s AI Overviews also provided inaccurate and potentially dangerous health information. This suggests a systemic problem with the application of large language models to complex medical domains.

Interestingly, the study also found that ChatGPT Health frequently over-triaged, advising 64 percent of individuals who did not require immediate care to go to the emergency room. While less immediately dangerous than under-triage, this could still contribute to overcrowding and strain on healthcare resources.

OpenAI’s Response and Potential Liability

OpenAI maintains that the study misinterpreted how people use ChatGPT Health and that the company is continuously working to improve its AI models. However, the independent evaluation raises serious concerns about the tool’s safety and reliability. The company has already faced accusations of its chatbot contributing to suicides and paranoid behavior, a phenomenon dubbed “AI psychosis.” Actively encouraging users to seek health advice – while simultaneously disclaiming responsibility for diagnosis or treatment – could expose OpenAI to significant legal liability.

The current disclaimer – that ChatGPT Health is “not intended for diagnosis or treatment” – feels increasingly inadequate given the tool’s demonstrated inability to accurately assess medical emergencies. It raises the question of whether offering such a flawed tool, even with a disclaimer, constitutes negligence.

What Does This Mean for Patients?

The findings from Nature Medicine are a clear reminder that AI chatbots are not a substitute for professional medical care. While these tools may offer convenience and access to information, they should not be relied upon for critical health decisions. If you are experiencing a medical emergency, it is crucial to seek immediate attention from a qualified healthcare provider.

It’s also important to remember that AI models are trained on data, and that data can contain biases and inaccuracies. The responses generated by these tools should always be viewed with a critical eye, and patients should never hesitate to seek a second opinion from a human doctor.

Looking Ahead: OpenAI has not announced specific changes to ChatGPT Health in response to the study. However, it is likely that the company will need to address the identified shortcomings to mitigate potential risks and maintain public trust. Further research and independent evaluations will be crucial to assess the safety and effectiveness of AI-powered health tools as they continue to evolve.

For reliable health information, consult official sources such as the Centers for Disease Control and Prevention (CDC) and the World Health Organization (WHO). And, most importantly, always prioritize a consultation with a qualified healthcare professional for any health concerns.