---
title: "Data leakage into the benchmarks (research idea)"
description: "Data leakage appears on 16 recent papers and memorization on 3, almost all in AI journals, while thousands of papers report results on public benchmarks that language models may have seen in training. How much of reported progress is contamination, and can finance and economics benchmarks be protected?"
url: https://scholar4ai.com/ideas/data-leakage-in-benchmarks
markdown_url: https://scholar4ai.com/ideas/data-leakage-in-benchmarks.md
license: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/)
share_text: "Data leakage appears on 16 recent papers and memorization on 3, almost all in AI journals, while thousands of papers report results on public benchmarks that language models may have seen in training."
date: "2026-10-23"
updated: "2026-10-23"
status: "seed"
tags: ["ai", "methods"]
keywords: ["data leakage", "memorization"]
---

# Data leakage into the benchmarks

Data leakage appears on 16 recent papers and memorization on 3, almost all in AI journals, while thousands of papers report results on public benchmarks that language models may have seen in training. How much of reported progress is contamination, and can finance and economics benchmarks be protected?

## What prompted this

Since October 2025, 29 papers carry a data-leakage, memorization, contamination or benchmark-leakage keyword (23 in AI journals): "data leakage" on 16, "memorization" on 3, "training data leakage" on 2. Benchmark results, by contrast, appear in a large share of the 17,000 AI-journal papers in the archive, and in the finance papers that test language models on public financial question sets.

## Questions

- For the benchmarks most used in the AI journals, how much do scores fall on freshly written items of the same kind, and does the fall differ between open and closed models?
- Have the financial and economic test sets used to evaluate language models (exam questions, sentiment sets) been published online before the models' training cutoffs?
- Can rolling benchmarks built from new filings, news and data releases give uncontaminated tests, and would journals require them?

## Data that could help

The release dates of benchmark items and of model training cutoffs, fresh item sets written after the cutoffs, and the results tables of recent papers.

## Next step

Rewrite 200 items from a widely used financial question benchmark with the same structure and new numbers, and compare model scores on the old and new items.

## Related research activity

*The keywords this note follows, in the tracked finance, accounting, economics, management, operations research and AI journals as of 2026-10-26*

| Keyword | Papers, last 30 days | Rank | Papers per month, Oct 23 – Oct 26* | Papers, last 36 months |
|---|---:|---:|---|---:|
| data leakage | 2 | #1303 | 0, 2, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 2, 1, 1, 2, 2, 4, 3, 0, 0, 1, 2, 0, 3, 0, 2 | 34 |
| memorization | 0 | – | 0, 0, 0, 1, 0, 0, 3, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 2, 2, 0, 0, 0, 1, 0, 1, 0, 0, 0 | 15 |

## Cite this page

Scholar4AI. (2026). Data leakage into the benchmarks. Journal Research Rankings. Retrieved October 23, 2026, from https://scholar4ai.com/ideas/data-leakage-in-benchmarks

```bibtex
@misc{scholar4ai-ideas-data-leakage-in-benchmarks,
  author = {{Scholar4AI}},
  title = {Data leakage into the benchmarks},
  year = {2026},
  howpublished = {\url{https://scholar4ai.com/ideas/data-leakage-in-benchmarks}},
  urldate = {2026-10-23},
  note = {Journal Research Rankings; data from OpenAlex (CC0); licensed CC BY-NC 4.0}
}
```

License: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/).