Radiologists & AI Struggle to Spot Deepfake X-rays: New Study

The increasing sophistication of artificial intelligence is now extending to medical imaging, with concerning implications for diagnostic accuracy and patient safety. A new study, published March 24 in Radiology, reveals that both experienced radiologists and advanced AI systems are struggling to differentiate between genuine X-rays and remarkably realistic “deepfake” images generated by AI. This blurring of lines between real and synthetic raises critical questions about the integrity of medical records and the potential for misuse, from fraudulent claims to deliberate clinical disruption.

The Challenge of Detecting Synthetic Images

Deepfakes, as the term suggests, are media – videos, photos, images, or audio – that appear authentic but have been created or altered using artificial intelligence. In the context of medical imaging, So AI can now generate X-rays that are visually indistinguishable from those produced by traditional methods. The study, led by Dr. Mickael Tordjman of the Icahn School of Medicine at Mount Sinai in New York, investigated how well radiologists could identify these AI-generated images.

Seventeen radiologists from twelve institutions across six countries – the United States, France, Germany, Turkey, the United Kingdom, and the United Arab Emirates – participated in the research. Their experience levels varied widely, ranging from those new to the field to seasoned specialists with up to 40 years of practice. Participants were presented with 264 X-ray images, half real and half synthetic. The AI-generated images were created using two different models: ChatGPT and RoentGen, an open-source generative AI diffusion model developed by Stanford Medicine researchers. RSNA News provides further details on the study’s methodology.

Accuracy Rates and the Role of Awareness

The results were striking. When radiologists were not informed that the image set included deepfakes, only 41% correctly identified the AI-generated X-rays based on their technical quality. Yet, when explicitly told to look for synthetic images, their accuracy improved to 75%. This suggests that awareness of the possibility of deepfakes can heighten scrutiny, but doesn’t guarantee accurate detection.

Individual performance varied considerably, with radiologists correctly identifying between 58% and 92% of the ChatGPT-generated images. Interestingly, even advanced AI systems struggled with the task. Four multimodal large language models – GPT-4o (OpenAI), GPT-5 (OpenAI), Gemini 2.5 Pro (Google), and Llama 4 Maverick (Meta) – achieved accuracy rates ranging from 57% to 85%. Notably, ChatGPT-4o, the very model used to create some of the deepfakes, wasn’t able to identify all of its own creations, though it did outperform the other models tested. News-Medical.net highlights this finding.

Subspecialty and Experience: Do They Matter?

The study also explored whether a radiologist’s experience level influenced their ability to spot deepfakes. Surprisingly, there was no correlation between years of practice and detection accuracy. However, radiologists specializing in musculoskeletal imaging demonstrated significantly better performance than those in other subspecialties. This suggests that familiarity with the subtle nuances of bone structure and related imaging characteristics may provide an advantage.

Visual Clues and the “Too Perfect” Image

Researchers identified several patterns that tend to appear in synthetic images, offering potential clues for detection. According to Dr. Tordjman, “Deepfake medical images often look too perfect.” Specifically, bones may appear overly smooth, spines unnaturally straight, lungs excessively symmetrical, blood vessel patterns unusually uniform, and fractures may seem unusually clean and consistent, often limited to one side of the bone. ScienceDaily illustrates these differences with comparative images of real and AI-generated radiographs.

Potential Risks and the Need for Safeguards

The implications of these findings are far-reaching. The potential for misuse of deepfake X-rays is substantial, ranging from fraudulent litigation – where fabricated fractures could be presented as evidence – to deliberate cybersecurity attacks. Hackers could potentially inject synthetic images into hospital networks to manipulate patient diagnoses or cause widespread clinical chaos by undermining trust in the reliability of digital medical records.

To mitigate these risks, researchers are advocating for stronger digital protections. These include embedding invisible watermarks directly into images and utilizing cryptographic signatures linked to the technologist at the time of image capture. These measures would help verify the authenticity and integrity of medical images.

Looking Ahead: The Evolution of AI in Medical Imaging

The current study focused on X-rays, but researchers anticipate that AI will soon be capable of generating convincing synthetic images for other modalities, such as CT and MRI scans. Dr. Tordjman notes that “We are potentially only seeing the tip of the iceberg.” He emphasizes the critical need to establish educational datasets and develop robust detection tools now, to prepare for this evolving landscape.

To facilitate education and awareness, the research team has released a curated deepfake dataset, complete with interactive quizzes designed for training purposes. This resource will be invaluable for healthcare professionals seeking to enhance their ability to identify synthetic medical images.

Ongoing Research and Educational Initiatives

The Radiological Society of North America (RSNA) is actively promoting research and educational initiatives to address the challenges posed by deepfake medical imaging. This includes developing new algorithms for detecting synthetic images and creating training programs for radiologists and other healthcare professionals. The focus is on equipping the medical community with the tools and knowledge necessary to navigate this emerging threat and maintain the highest standards of patient care.