Benchmark inertia in AI journals
The dataset ranking shows CIFAR and MNIST still among the most-mentioned image benchmarks in AI journal papers, while cross-dataset generalization and reproducibility are rising keywords. Do gains on these old benchmarks carry over to harder, shifted data?
What prompted this
On the dataset ranking, CIFAR-10/100 and MNIST are each named in the titles or abstracts of 7 AI journal papers in the 30 days to 2026-10-23, more than ImageNet (3). On the AI dashboard, "cross-dataset generalization" (24 papers in July–September 2026 against about 13 expected) and "reproducibility" (15 against 7) are growing. Since October 2025, 385 papers carry a benchmark keyword (287 in AI journals), and 28 of them also carry a reproducibility or generalization keyword.
Questions
- For methods published with gains on CIFAR or MNIST, how much of the gain survives on newer or distribution-shifted datasets?
- How much of a reported gain comes from tuning effort rather than from the method?
- Should journals ask for at least one harder benchmark per paper?
Possible approach
Rerun released code from recent papers on a shifted benchmark, such as one of the domain-adaptation suites, with the same tuning budget for the method and its baselines.
Next step
Take ten recent AI-journal papers that report CIFAR gains and release their code, and rerun them on one distribution-shift benchmark.