Model collapse and the data that is left
Synthetic data generation appears on 77 recent papers and synthetic data on 18, almost all in AI journals, while model collapse, the degradation of models trained on their predecessors' output, appears on 3. The web is filling with generated text and images that will train the next models. How fast does quality decay, and what does that do to the value of human data?
What prompted this
Since October 2025, 117 papers carry a synthetic-data, model-collapse, self-consuming or recursive-training keyword (111 in AI journals): "synthetic data generation" on 77, "synthetic data" on 18, "synthetic data augmentation" on 10, "model collapse" on 3. Eleven of the 117 also carry a collapse, degradation, recursive, distribution-shift or contamination keyword, and "data quality" appears on 30 papers in the archive.
Questions
- When models are trained on mixtures of human and generated data at the shares now seen on the web, how fast do rare patterns and tails disappear, and does filtering by quality stop it?
- Does the decay differ for text, images and tabular data used in finance and economics?
- If human-written data becomes scarce, what is it worth, and who owns the archives that have it?
Data that could help
Web crawl samples by year with AI-text detection, controlled training runs on mixed data, and licensing deals for data archives.
Next step
Train a small model on successive generations of mixed human and generated data at web-realistic shares and measure the loss of tail vocabulary and rare facts per generation.