---
title: "Pre-training corpora (Common Crawl, C4, The Pile)"
description: "Pre-training corpora (Common Crawl, C4, The Pile): compare tasks, official access, usage conditions and evaluation. Includes research-paper mention counts and trends."
url: https://scholar4ai.com/datasets/pre-training-corpora-common-crawl-c4-the-pile
markdown_url: https://scholar4ai.com/datasets/pre-training-corpora-common-crawl-c4-the-pile.md
license: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/)
share_text: "Pre-training corpora (Common Crawl, C4, The Pile): compare tasks, official access, usage conditions and evaluation. Includes research-paper mention counts and trends. (as of 2026-11-05)"
data_url: https://scholar4ai.com/data/rankings.json
updated: 2026-11-05T06:02:37+09:00
window: 2026-10-07 to 2026-11-05
source: OpenAlex (CC0)
---

# Pre-training corpora (Common Crawl, C4, The Pile)

[Browse dataset profiles](https://scholar4ai.com/datasets/directory/page/5.md)

Corpora and benchmarks for language understanding, question answering and text classification.

- data category: Text and NLP benchmarks
- 0 papers in the last 7 days
- Names searched for: Common Crawl, CommonCrawl, C4 corpus, BookCorpus, RefinedWeb

Most-used data: https://scholar4ai.com/rankings/datasets.md · All Text and NLP benchmarks datasets: https://scholar4ai.com/datasets/text-and-nlp-benchmarks.md

## Tasks, access and usage conditions

Mention counts below measure name matches in tracked papers, not benchmark quality or downloads. A family count can cover multiple datasets; it is not a separate count for each comparison row. Choose a task and version first, preserve evaluation splits, and follow the linked release terms. The site's content license does not replace the datasets' own terms.

*NLP dataset comparison: tasks, access and usage conditions*

| Dataset / version | Task and evaluation | Access and preparation | Usage conditions |
|---|---|---|---|
| Common Crawl | Raw web crawl corpus used to build pretraining datasets. Evaluation: Evaluate the trained model on held-out tasks; document crawl date and contamination checks, not just corpus size. | [Official source / documentation](https://commoncrawl.org/get-started) Select a dated crawl and WARC/WAT/WET files; filtering, deduplication and language selection are additional work. | Common Crawl’s terms of use apply; access does not remove third-party copyright, privacy or other source-content obligations. |
| C4 | Filtered Common Crawl text for language-model pretraining. Evaluation: Document filtering, language and deduplication; evaluate contamination against downstream benchmarks. | [Official source / documentation](https://huggingface.co/datasets/allenai/c4) Select a named version/configuration and streaming or shard downloads; English, multilingual and other variants are different corpora. | The AllenAI distribution declares ODC-By; source web-text rights still apply. This is distinct from the Apache-2.0 license of dataset-loading code. |
| The Pile | A mixture of multiple text sources for language-model pretraining. Evaluation: Report component weighting, exclusions and held-out evaluation. This tracked family’s search aliases also include other web corpora, so its mention count is not a Pile-only count. | [Official source / documentation](https://pile.eleuther.ai/) Use the project’s component documentation and available authorized releases; identify the exact mixture and version. | Check component-specific licenses and source rights; a repository code license is not a universal license over every text component. |

Guide updated 2026-11-04.

## Key findings

- From Nov 2023 to Oct 2026, 108 archived papers mentioned Pre-training corpora (Common Crawl, C4, The Pile) in their title, abstract or keywords (the search of earlier months' full texts is still under way), most of them in Aug 2024 (9).
- No paper of the tracked journals mentioned Pre-training corpora (Common Crawl, C4, The Pile) in the last 30 days (2026-10-07 to 2026-11-05).
- From 2017 to 2026 (so far), Pre-training corpora (Common Crawl, C4, The Pile) was mentioned in 110 papers in the tracked journals, most of them in AI (95), information systems (6) and higher education and research (4).
- 38 of them were published in 2025, against 0 in 2017.
- Since 2023, the institutions with the most of these papers were Carnegie Mellon University (4), Centre National de la Recherche Scientifique (3) and Chinese Academy of Sciences (3).
- The journals publishing most of them were Artificial Intelligence Review (13), AI & Society (12) and Language Resources and Evaluation (8).
- The most-cited of them is “Vision-Language Models for Vision Tasks: A Survey” (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024), with 998 citations.
- Papers use Pre-training corpora (Common Crawl, C4, The Pile) most often with Wikipedia (75), Hugging Face Hub (42) and Twitter / X (35); it has 30 partners in the code and data network, in the group around Twitter / X, Facebook, YouTube / TikTok.

## Mentions over 36 months

*Papers mentioning it by publication month, Nov 2023 to Oct 2026 (titles, abstracts and keywords; the full texts of earlier months are still being searched)*

| Trend | Nov 23–Oct 24 | Nov 24–Oct 25 | Nov 25–Oct 26 | 36 months |
|---|---:|---:|---:|---:|
|  | 41 | 45 | 22 | 108 |

## Papers per year

*Papers per publication year and category*

| Category | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026 (so far) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| [AI](https://scholar4ai.com/categories/ai.md) | 0 | 0 | 0 | 1 | 0 | 0 | 6 | 37 | 34 | 17 |
| [Information Systems](https://scholar4ai.com/categories/information-systems.md) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 1 | 1 |
| [Higher Education and Research](https://scholar4ai.com/categories/higher-education-and-research.md) | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 2 | 1 |
| [Management](https://scholar4ai.com/categories/management.md) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 1 | 0 |
| [Economics](https://scholar4ai.com/categories/economics.md) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 |
| All categories | 0 | 0 | 0 | 1 | 0 | 0 | 7 | 44 | 38 | 20 |

## Where the papers come from

*Journals with the most papers, since 2023*

| # | Journal | Papers |
|---:|---|---:|
| 1 | [Artificial Intelligence Review](https://scholar4ai.com/journals/artificial-intelligence-review.md) | 13 |
| 2 | [AI & Society](https://scholar4ai.com/journals/ai-society.md) | 12 |
| 3 | [Language Resources and Evaluation](https://scholar4ai.com/journals/language-resources-and-evaluation.md) | 8 |
| 4 | [IEEE Transactions on Pattern Analysis and Machine Intelligence](https://scholar4ai.com/journals/ieee-transactions-on-pattern-analysis-and-machine-intelligence.md) | 7 |
| 5 | [Neural Computing and Applications](https://scholar4ai.com/journals/neural-computing-and-applications.md) | 7 |
| 6 | [Computer Speech & Language](https://scholar4ai.com/journals/computer-speech-language.md) | 6 |
| 7 | [Neural Networks](https://scholar4ai.com/journals/neural-networks.md) | 6 |
| 8 | [Expert Systems with Applications](https://scholar4ai.com/journals/expert-systems-with-applications.md) | 5 |
| 9 | [Machine Learning](https://scholar4ai.com/journals/machine-learning.md) | 4 |
| 10 | [Information Processing & Management](https://scholar4ai.com/journals/information-processing-management.md) | 3 |

*Institutions with the most papers, since 2023*

| # | Institution | Papers |
|---:|---|---:|
| 1 | [Carnegie Mellon University](https://scholar4ai.com/institutions/carnegie-mellon-university.md) | 4 |
| 2 | [Centre National de la Recherche Scientifique](https://scholar4ai.com/institutions/centre-national-de-la-recherche-scientifique.md) | 3 |
| 3 | [Chinese Academy of Sciences](https://scholar4ai.com/institutions/chinese-academy-of-sciences.md) | 3 |
| 4 | Halmstad University | 3 |
| 5 | [Nanyang Technological University](https://scholar4ai.com/institutions/nanyang-technological-university.md) | 3 |
| 6 | [University of Chinese Academy of Sciences](https://scholar4ai.com/institutions/university-of-chinese-academy-of-sciences.md) | 3 |
| 7 | [University of Pisa](https://scholar4ai.com/institutions/university-of-pisa.md) | 3 |
| 8 | AgroParisTech | 2 |
| 9 | Centre de Coopération Internationale en Recherche Agronomique pour le Développement | 2 |
| 10 | [Hefei University of Technology](https://scholar4ai.com/institutions/hefei-university-of-technology.md) | 2 |

*Countries with the most papers, since 2023*

| # | Country | Papers |
|---:|---|---:|
| 1 | China | 21 |
| 2 | United Kingdom | 13 |
| 3 | United States | 13 |
| 4 | Germany | 12 |
| 5 | Italy | 11 |
| 6 | Australia | 7 |
| 7 | Spain | 7 |
| 8 | France | 7 |
| 9 | India | 7 |
| 10 | Hong Kong | 6 |

*Keywords with the most papers, since 2023*

| # | Keyword | Papers |
|---:|---|---:|
| 1 | [large language models](https://scholar4ai.com/keywords/large-language-models.md) | 37 |
| 2 | fine-tuning | 12 |
| 3 | low-resource languages | 10 |
| 4 | [BERT](https://scholar4ai.com/keywords/bert.md) | 8 |
| 5 | [ChatGPT](https://scholar4ai.com/keywords/chatgpt.md) | 8 |
| 6 | [generative AI](https://scholar4ai.com/keywords/generative-ai.md) | 8 |
| 7 | language models | 7 |
| 8 | prompt engineering | 7 |
| 9 | [few-shot learning](https://scholar4ai.com/keywords/few-shot-learning.md) | 5 |
| 10 | [sentiment analysis](https://scholar4ai.com/keywords/sentiment-analysis.md) | 5 |

## Most-cited papers

*Most-cited papers since 2017*

| # | Citations | Per month | Title | Journal | Published | Authors |
|---:|---:|---:|---|---|---|---|
| 1 | 998 | 30.9 | [Vision-Language Models for Vision Tasks: A Survey](https://doi.org/10.1109/tpami.2024.3369699) | [IEEE Transactions on Pattern Analysis and Machine Intelligence](https://scholar4ai.com/journals/ieee-transactions-on-pattern-analysis-and-machine-intelligence.md) | 2024-02-26 | Jingyi Zhang, Jiaxing Huang, Sheng Jin et al. |
| 2 | 395 | 14.9 | [Large language models (LLMs): survey, technical frameworks, and future challenges](https://doi.org/10.1007/s10462-024-10888-y) | [Artificial Intelligence Review](https://scholar4ai.com/journals/artificial-intelligence-review.md) | 2024-08-18 | Pranjal Kumar |
| 3 | 378 | 11.5 | [Leveraging large language models for predictive chemistry](https://doi.org/10.1038/s42256-023-00788-1) | [Nature Machine Intelligence](https://scholar4ai.com/journals/nature-machine-intelligence.md) | 2024-02-06 | Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega‐Guerrero et al. |
| 4 | 312 | 9.0 | [PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting](https://doi.org/10.1109/tkde.2023.3342137) | [IEEE Transactions on Knowledge and Data Engineering](https://scholar4ai.com/journals/ieee-transactions-on-knowledge-and-data-engineering.md) | 2023-12-13 | Hao Xue, Flora Dilys Salim |
| 5 | 209 | 6.1 | [Students’ perceptions of using ChatGPT in a physics class as a virtual tutor](https://doi.org/10.1186/s41239-023-00434-1) | [International Journal of Educational Technology in Higher Education](https://scholar4ai.com/journals/international-journal-of-educational-technology-in-higher-education.md) | 2023-12-21 | Lu Ding, Tong Li, Shiyan Jiang et al. |

## Used together with

Papers use Pre-training corpora (Common Crawl, C4, The Pile) most often with Wikipedia (75), Hugging Face Hub (42) and Twitter / X (35); it has 30 partners in the code and data network, in the group around Twitter / X, Facebook, YouTube / TikTok.

*What papers use Pre-training corpora (Common Crawl, C4, The Pile) together with (at least 3 papers; lift 1 = chance)*

| Used with | Type | Papers using both | Lift |
|---|---|---:|---:|
| [Wikipedia](https://scholar4ai.com/datasets/wikipedia.md) | Dataset | 75 | 32.1 |
| [Hugging Face Hub](https://scholar4ai.com/datasets/hugging-face-hub.md) | Dataset | 42 | 39.1 |
| [Twitter / X](https://scholar4ai.com/datasets/twitter-x.md) | Dataset | 35 | 3.4 |
| [Facebook](https://scholar4ai.com/datasets/facebook.md) | Dataset | 31 | 4.5 |
| [Reddit](https://scholar4ai.com/datasets/reddit.md) | Dataset | 20 | 14.4 |
| [ImageNet](https://scholar4ai.com/datasets/imagenet.md) | Dataset | 16 | 3.0 |
| [YouTube / TikTok](https://scholar4ai.com/datasets/youtube-tiktok.md) | Dataset | 13 | 3.1 |
| [CoNLL](https://scholar4ai.com/datasets/conll.md) | Dataset | 9 | 30.8 |
| [MMLU](https://scholar4ai.com/datasets/mmlu.md) | Dataset | 9 | 49.5 |
| [TruthfulQA / HellaSwag / ARC](https://scholar4ai.com/datasets/truthfulqa-hellaswag-arc.md) | Dataset | 9 | 55.9 |
| [GLUE / SuperGLUE](https://scholar4ai.com/datasets/glue-superglue.md) | Dataset | 8 | 32.2 |
| [Natural language inference (SNLI, MultiNLI)](https://scholar4ai.com/datasets/natural-language-inference-snli-multinli.md) | Dataset | 8 | 37.0 |

The whole network: https://scholar4ai.com/code/network.md

## Cite this page

Scholar4AI. (2026). Pre-training corpora (Common Crawl, C4, The Pile). Journal and Conference Research Rankings. Retrieved November 5, 2026, from https://scholar4ai.com/datasets/pre-training-corpora-common-crawl-c4-the-pile

```bibtex
@misc{scholar4ai-datasets-pre-training-corpora-common-crawl-c4-the-pile,
  author = {{Scholar4AI}},
  title = {Pre-training corpora (Common Crawl, C4, The Pile)},
  year = {2026},
  howpublished = {\url{https://scholar4ai.com/datasets/pre-training-corpora-common-crawl-c4-the-pile}},
  urldate = {2026-11-05},
  note = {Journal and Conference Research Rankings; data from OpenAlex (CC0); licensed CC BY-NC 4.0}
}
```

License: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/).