Bad Data in AI Research: Provenance & The Five Safes Framework

The medical research landscape is facing a reckoning with data integrity, highlighted by a recent case detailed in the Journal of Medical Internet Research. A flawed dataset, initially uploaded to the popular platform Kaggle, infiltrated the scientific record, appearing in over 90 published papers before being flagged and triggering a wave of retractions. This incident underscores a growing vulnerability: the speed and scale at which machine learning algorithms can amplify misinformation when trained on unvalidated data, potentially harming vulnerable populations and eroding trust in science.

The dataset in question comprised unverified images of children sourced from websites related to autism, intended for use in an artificial intelligence model designed to “detect the presence of autism or the absence thereof.” The fundamental flaw – the lack of validation and ethical sourcing of the images – went unnoticed until a reviewer identified the problem during the publication process in December 2025. This incident isn’t isolated, but rather a symptom of broader challenges in data governance as open access datasets turn into increasingly central to modern research.

The Expanding Role of Open Data and Machine Learning

Platforms like Kaggle and GitHub have become invaluable resources for data scientists and software developers, offering free access to datasets for training machine learning algorithms. However, as Alan Katz, a professor of family medicine at the Manitoba Centre for Health Policy (MCHP), points out, these platforms often lack the rigorous documentation, governance, and quality control measures found in established medical databases like the MCHP’s own. Katz described the situation as “both shocking, but also not surprising” given the rapid expansion of open access databases and their use in AI research. The MCHP, for example, employs dedicated staff to validate all new data before it’s made available.

The problem isn’t necessarily the existence of open data, argues Elizabeth Green, a lecturer in business and law at the University of the West of England, Bristol. She cites DermAtlas, an open-source medical database of skin conditions, as a “fantastic resource” particularly helpful in diagnosing rare cases. Instead, the focus should be on strengthening governance systems to balance the benefits of open access with the risks of flawed data. This is particularly critical as machine learning allows for analyses to be generated and published at an unprecedented rate, accelerating the propagation of errors throughout the research ecosystem.

The Five Safes Framework: A Potential Solution

One promising approach to bolstering data integrity is the Five Safes framework, developed by Felix Ritchie. This framework provides a structured approach to evaluating data through five key areas:

1. Safe Project: Ensuring data is ethically collected and clinically validated by experts.
2. Safe People: Confirming researchers accessing the data are qualified and specifically trained in using AI-based datasets.
3. Safe Data: Independently validating the data and tracking all accesses and modifications.
4. Safe Settings: Verifying that health data was acquired in a clinical setting and is securely stored.
5. Safe Outputs: Confirming valid methodologies and statistics were used to derive the results.

Ritchie believes that implementing the Five Safes could form the backbone of a data provenance system, requiring compliance before a manuscript is considered for publication. Australia has already legislated the framework, demonstrating a growing recognition of its importance. A potential workflow could involve third-party certification of data, storage in a secure registry potentially leveraging blockchain technology (similar to that used for financial transactions), and mandatory ethical approval and data security certification for submitted manuscripts. Xcelligen highlights the role of machine learning in automating these processes, detecting anomalies, and ensuring regulatory compliance.

Who Bears the Responsibility?

Establishing data integrity isn’t the responsibility of a single entity. It requires a collaborative effort across multiple stakeholders. Data-sharing platforms, research institutions, funding bodies, and academic publishers all play a crucial role in ensuring data is shared, vetted, and incorporated into the scientific record responsibly. Funding bodies, for instance, already often tie grants to ethical research standards, as is the case in Canada, according to Katz.

Anne Borden, an autism advocate and journalist, emphasizes the urgency of addressing this issue. “You really have to stop misinformation being perpetuated under the banner of science,” she says, “because once it’s out there, you’re done. The Internet is forever.” The incident serves as a stark reminder of the potential consequences of unchecked data quality in the age of rapid AI development.

The Broader Implications for Machine Learning Governance

This case extends beyond a single flawed dataset. It highlights the need for more robust machine learning data governance practices. As outlined in a report by AIMultiple, machine learning data governance encompasses the policies, processes, and technologies needed to manage and utilize data effectively in machine learning applications. This includes not only data quality but also data security, privacy, and compliance with relevant regulations.

The Journal of Medical Internet Research article points to the need for enforced data provenance systems. Without a clear understanding of where data originated and how it has been processed, it’s difficult to assess its reliability and validity. This is particularly important in sensitive areas like medical research, where inaccurate data can have serious consequences. Data Governance Platforms notes that machine learning algorithms are increasingly being used to enhance data quality, but these algorithms themselves require high-quality, validated data to function effectively.

Looking Ahead: Towards a More Robust Data Ecosystem

The path forward involves a multi-faceted approach. Implementing frameworks like the Five Safes, strengthening data validation processes at data-sharing platforms, and fostering a culture of data integrity within research institutions are all essential steps. Ritchie envisions a “register of validated, ethical datasets” as a potential game-changer, providing researchers with a trusted source of high-quality data.

the goal is to harness the transformative potential of machine learning while mitigating the risks associated with flawed data. This requires a commitment to proactive data governance, ethical data practices, and a shared responsibility among all stakeholders in the research ecosystem. The recent incident serves as a critical learning opportunity, prompting a necessary self-reflection and correction to ensure the continued integrity and trustworthiness of medical research.