Pre-training corpora (Common Crawl, C4, The Pile)
← Most-used data · All Text and NLP benchmarks datasets
Corpora and benchmarks for language understanding, question answering and text classification.
Tasks, access and usage conditions#
Mention counts below measure name matches in tracked papers, not benchmark quality or downloads. A family count can cover multiple datasets; it is not a separate count for each comparison row. Choose a task and version first, preserve evaluation splits, and follow the linked release terms. The site's content license does not replace the datasets' own terms.
| Dataset / version | Task and evaluation | Access and preparation | Usage conditions |
|---|---|---|---|
| Common Crawl | Raw web crawl corpus used to build pretraining datasets. Evaluation: Evaluate the trained model on held-out tasks; document crawl date and contamination checks, not just corpus size. | Official source / documentation Select a dated crawl and WARC/WAT/WET files; filtering, deduplication and language selection are additional work. | Common Crawl’s terms of use apply; access does not remove third-party copyright, privacy or other source-content obligations. |
| C4 | Filtered Common Crawl text for language-model pretraining. Evaluation: Document filtering, language and deduplication; evaluate contamination against downstream benchmarks. | Official source / documentation Select a named version/configuration and streaming or shard downloads; English, multilingual and other variants are different corpora. | The AllenAI distribution declares ODC-By; source web-text rights still apply. This is distinct from the Apache-2.0 license of dataset-loading code. |
| The Pile | A mixture of multiple text sources for language-model pretraining. Evaluation: Report component weighting, exclusions and held-out evaluation. This tracked family’s search aliases also include other web corpora, so its mention count is not a Pile-only count. | Official source / documentation Use the project’s component documentation and available authorized releases; identify the exact mixture and version. | Check component-specific licenses and source rights; a repository code license is not a universal license over every text component. |
Key findings#
- From Nov 2023 to Oct 2026, 108 archived papers mentioned Pre-training corpora (Common Crawl, C4, The Pile) in their title, abstract or keywords (the search of earlier months' full texts is still under way), most of them in Aug 2024 (9).
- No paper of the tracked journals mentioned Pre-training corpora (Common Crawl, C4, The Pile) in the last 30 days (2026-10-06 to 2026-11-04).
- From 2017 to 2026 (so far), Pre-training corpora (Common Crawl, C4, The Pile) was mentioned in 110 papers in the tracked journals, most of them in AI (95), information systems (6) and higher education and research (4).
- 38 of them were published in 2025, against 0 in 2017.
- Since 2023, the institutions with the most of these papers were Carnegie Mellon University (4), Centre National de la Recherche Scientifique (3) and Chinese Academy of Sciences (3).
- The journals publishing most of them were Artificial Intelligence Review (13), AI & Society (12) and Language Resources and Evaluation (8).
- The most-cited of them is “Vision-Language Models for Vision Tasks: A Survey” (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024), with 998 citations.
- Papers use Pre-training corpora (Common Crawl, C4, The Pile) most often with Wikipedia (75), Hugging Face Hub (42) and Twitter / X (35); it has 30 partners in the code and data network, in the group around Twitter / X, Facebook, YouTube / TikTok.
Mentions over 36 months#
| Trend | Nov 23–Oct 24 | Nov 24–Oct 25 | Nov 25–Oct 26 | 36 months |
|---|---|---|---|---|
| 41 | 45 | 22 | 108 |
Papers per year#
| Category | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026 (so far) |
|---|---|---|---|---|---|---|---|---|---|---|
| AI | 0 | 0 | 0 | 1 | 0 | 0 | 6 | 37 | 34 | 17 |
| Information Systems | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 1 | 1 |
| Higher Education and Research | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 2 | 1 |
| Management | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 1 | 0 |
| Economics | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 |
| All categories | 0 | 0 | 0 | 1 | 0 | 0 | 7 | 44 | 38 | 20 |
Where the papers come from#
| # | Journal | Papers |
|---|---|---|
| 1 | Artificial Intelligence Review | 13 |
| 2 | AI & Society | 12 |
| 3 | Language Resources and Evaluation | 8 |
| 4 | IEEE Transactions on Pattern Analysis and Machine Intelligence | 7 |
| 5 | Neural Computing and Applications | 7 |
| 6 | Computer Speech & Language | 6 |
| 7 | Neural Networks | 6 |
| 8 | Expert Systems with Applications | 5 |
| 9 | Machine Learning | 4 |
| 10 | Information Processing & Management | 3 |
| # | Institution | Papers |
|---|---|---|
| 1 | Carnegie Mellon University | 4 |
| 2 | Centre National de la Recherche Scientifique | 3 |
| 3 | Chinese Academy of Sciences | 3 |
| 4 | Halmstad University | 3 |
| 5 | Nanyang Technological University | 3 |
| 6 | University of Chinese Academy of Sciences | 3 |
| 7 | University of Pisa | 3 |
| 8 | AgroParisTech | 2 |
| 9 | Centre de Coopération Internationale en Recherche Agronomique pour le Développement | 2 |
| 10 | Hefei University of Technology | 2 |
| # | Country | Papers |
|---|---|---|
| 1 | China | 21 |
| 2 | United Kingdom | 13 |
| 3 | United States | 13 |
| 4 | Germany | 12 |
| 5 | Italy | 11 |
| 6 | Australia | 7 |
| 7 | Spain | 7 |
| 8 | France | 7 |
| 9 | India | 7 |
| 10 | Hong Kong | 6 |
| # | Keyword | Papers |
|---|---|---|
| 1 | large language models | 37 |
| 2 | fine-tuning | 12 |
| 3 | low-resource languages | 10 |
| 4 | BERT | 8 |
| 5 | ChatGPT | 8 |
| 6 | generative AI | 8 |
| 7 | language models | 7 |
| 8 | prompt engineering | 7 |
| 9 | few-shot learning | 5 |
| 10 | sentiment analysis | 5 |
Most-cited papers#
| # | Citations | Per month | Title | Journal | Published | Authors |
|---|---|---|---|---|---|---|
| 1 | 998 | 30.9 | Vision-Language Models for Vision Tasks: A Survey | IEEE Transactions on Pattern Analysis and Machine Intelligence | 2024-02-26 | Jingyi Zhang, Jiaxing Huang, Sheng Jin et al. |
| 2 | 395 | 14.9 | Large language models (LLMs): survey, technical frameworks, and future challenges | Artificial Intelligence Review | 2024-08-18 | Pranjal Kumar |
| 3 | 378 | 11.5 | Leveraging large language models for predictive chemistry | Nature Machine Intelligence | 2024-02-06 | Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega‐Guerrero et al. |
| 4 | 312 | 9.0 | PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting | IEEE Transactions on Knowledge and Data Engineering | 2023-12-13 | Hao Xue, Flora Dilys Salim |
| 5 | 209 | 6.1 | Students’ perceptions of using ChatGPT in a physics class as a virtual tutor | International Journal of Educational Technology in Higher Education | 2023-12-21 | Lu Ding, Tong Li, Shiyan Jiang et al. |
Used together with#
| Used with | Type | Papers using both | Lift |
|---|---|---|---|
| Wikipedia | Dataset | 75 | 32.1 |
| Hugging Face Hub | Dataset | 42 | 39.1 |
| Twitter / X | Dataset | 35 | 3.4 |
| Dataset | 31 | 4.5 | |
| Dataset | 20 | 14.4 | |
| ImageNet | Dataset | 16 | 3.0 |
| YouTube / TikTok | Dataset | 13 | 3.1 |
| CoNLL | Dataset | 9 | 30.8 |
| MMLU | Dataset | 9 | 49.5 |
| TruthfulQA / HellaSwag / ARC | Dataset | 9 | 55.9 |
| GLUE / SuperGLUE | Dataset | 8 | 32.2 |
| Natural language inference (SNLI, MultiNLI) | Dataset | 8 | 37.0 |
Cite this page
Scholar4AI. (2026). Pre-training corpora (Common Crawl, C4, The Pile). Journal and Conference Research Rankings. Retrieved November 4, 2026, from https://scholar4ai.com/datasets/pre-training-corpora-common-crawl-c4-the-pile
@misc{scholar4ai-datasets-pre-training-corpora-common-crawl-c4-the-pile,
author = {{Scholar4AI}},
title = {Pre-training corpora (Common Crawl, C4, The Pile)},
year = {2026},
howpublished = {\url{https://scholar4ai.com/datasets/pre-training-corpora-common-crawl-c4-the-pile}},
urldate = {2026-11-04},
note = {Journal and Conference Research Rankings; data from OpenAlex (CC0); licensed CC BY-NC 4.0}
}