Common Corpus: Open Data Set Powers Global AI Training & Avoids Copyright Issues

The sourcing of training data for large language models (LLMs) remains a significant point of contention in the rapidly evolving field of artificial intelligence. While many companies currently employ a “scrape everything” approach, gathering vast datasets from across the internet, the legal implications of this practice are still largely unresolved. A growing alternative focuses on utilizing materials already in the public domain or released under permissive licenses, allowing for LLM training without copyright concerns. However, the fragmented nature of these resources has historically hindered their effectiveness. Now, a project called the Common Corpus is aiming to change that, and has recently expanded its reach significantly.

Building a Foundation for Open LLMs

Created by the French startup Pleias and initially released just over a year ago, the Common Corpus provides a centralized, curated collection of permissively licensed data for training LLMs. The project’s core principle is to offer a clear alternative to the legally ambiguous practice of scraping copyrighted material. A recent press release from the AI Alliance details the key characteristics of the Common Corpus: it’s truly open, multilingual, diverse, and extensively curated. The initial release contained over 2 trillion tokens – a standard measure of training data volume – and has now grown to over 2.267 trillion tokens with a recent update.

This expansion isn’t just about size; it’s about global representation. The updated corpus now includes substantial additions of data from China, Japan, Korea, Brazil, India, Africa, and Southeast Asia. Specifically, the latest release boasts data for eight languages with over 10 billion tokens each (English, French, German, Spanish, Italian, Polish, Greek, and Latin) and 33 languages exceeding 1 billion tokens. This broader linguistic scope is crucial for developing LLMs that aren’t solely focused on Western languages, and perspectives.

What’s Inside the Common Corpus?

The Common Corpus is organized into five main categories: OpenGovernment, OpenCulture, OpenScience, OpenWeb, and OpenSource. OpenGovernment includes datasets like Finance Commons, containing financial documents from governmental and regulatory bodies, and Legal Commons, a collection of legal and administrative texts. OpenCulture focuses on cultural heritage data, including books and newspapers, many dating back to the 18th and 19th centuries. OpenScience primarily comprises publicly available academic and scientific publications, often in PDF format. OpenWeb incorporates datasets from sources like YouTube Commons (transcripts from public domain videos) and Stack Exchange. Finally, OpenSource consists of code collected from permissively licensed GitHub repositories.

The curation process is a key differentiator. Pleias has not only focused on licensing but as well on data quality. Spelling and formatting errors have been corrected in digitized texts, harmful and toxic content has been removed, and material deemed to have low educational value has also been excluded. This level of refinement is intended to produce LLMs that are not only legally compliant but also more reliable and less prone to generating problematic outputs.

Beyond Legal Compliance: Auditable and Secure Models

The benefits of the Common Corpus extend beyond simply avoiding copyright infringement. Due to the fact that of its carefully selected and curated data, it enables the training of LLMs that are fully auditable – meaning the provenance of the training data is clear and traceable. This transparency is increasingly important as concerns grow about the “black box” nature of many AI systems. As the original press release explains, the Common Corpus “exceeds the requirements of even the strictest regulations on AI training data, such as the EU AI Act.”

Pleias has taken steps to ensure compliance with the General Data Protection Regulation (GDPR) by developing custom procedures for removing personally identifiable information (PII) from multilingual data. This makes the Common Corpus a potentially ideal foundation for secure, enterprise-grade models. Models trained on this dataset are expected to be more resilient to the increasing regulatory scrutiny surrounding AI.

Addressing Toxicity and Promoting Open Source AI

Another significant advantage is the proactive removal of material with high “toxicity scores.” This pre-filtering helps to mitigate the risk of LLMs generating harmful or biased content, a persistent challenge in the field. The Common Corpus also facilitates the development of models compatible with the Open Source Initiative’s definition of open-source AI, which emphasizes unrestricted utilize and modification. This openness is particularly relevant for initiatives like the push within the European Union to create “public AI” systems, built on open-source software and accessible to all.

The French government is already supporting the project, alongside other organizations committed to openness. The corpus’s development has been a collaborative effort, involving the AI Alliance, the French Ministry of Culture, Wikimedia Enterprise, Wikidata/Wikimedia Germany, and Libraries Without Borders, with infrastructure support from Jean Zay (Eviden, Idris), Tracto AI, and Mozilla.

A Call for Broader Support

The Common Corpus represents a powerful demonstration of the benefits of openness and permissive copyright licensing in the context of AI development. It offers a viable alternative to the potentially problematic practice of indiscriminate data scraping. As such, it deserves wider support. More governments should consider backing the project as a means of fostering innovation and transparency. Publishers, too, would be well-advised to invest in the Common Corpus, as it provides a resource specifically designed to navigate the complex copyright issues that plague the generative AI landscape today.

The future of LLM training may well depend on initiatives like the Common Corpus – projects that prioritize ethical data sourcing, transparency, and collaboration. The continued expansion of the corpus, particularly in terms of linguistic diversity and data quality, will be crucial for unlocking the full potential of these powerful technologies. Further development will likely focus on refining the data curation process, expanding language support, and exploring new methods for ensuring data provenance and security.