The annotation labour behind AI
Annotation cost reduction and reinforcement learning from human feedback appear on 44 recent papers in the AI journals, among 71 papers on annotation and human feedback, but only 8 of them carry a worker, labour, wage, platform or ethics keyword. Millions of people label data and rate model outputs, often through platforms in low-income countries. What are they paid, and how does the pay shape the data?
What prompted this
Since October 2025, 71 papers carry an annotation-cost, human-annotation, annotator, crowdwork, human-feedback, RLHF or data-labelling keyword (69 in AI journals): "annotation cost reduction" on 27, "reinforcement learning from human feedback" on 17, "human-in-the-loop annotation" on 7, "expert annotation" on 5, "data annotation" on 6. Only 8 of the 71 also carry a worker, labour, wage, crowdwork, gig, pay, employment, economics, incentive, platform, fairness or ethics keyword; "ghost workers" and "precarious employment" appear on one and two papers.
Questions
- What do annotators and raters earn per hour on the main platforms by country and task, and how has pay changed as demand for preference ratings grew?
- Does pay level change label quality and the preferences that are encoded in models, for example toward speed over care?
- Where is annotation work moving, and do the new locations' labour laws cover it?
Data that could help
Platform task postings and pay rates, worker surveys, and experiments that vary pay and measure label quality.
Next step
Scrape posted pay rates for annotation and rating tasks on the main platforms over a year and relate them to task type and worker location.