---
title: "The annotation labour behind AI (research idea)"
description: "Annotation cost reduction and reinforcement learning from human feedback appear on 44 recent papers in the AI journals, among 71 papers on annotation and human feedback, but only 8 of them carry a worker, labour, wage, platform or ethics keyword. Millions of people label data and rate model outputs, often through platforms in low-income countries. What are they paid, and how does the pay shape the data?"
url: https://scholar4ai.com/ideas/annotation-labour-behind-ai
markdown_url: https://scholar4ai.com/ideas/annotation-labour-behind-ai.md
license: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/)
date: "2026-10-23"
updated: "2026-10-23"
status: "seed"
tags: ["ai", "labor", "ethics"]
keywords: ["annotation cost reduction", "reinforcement learning from human feedback", "data annotation"]
---

# The annotation labour behind AI

Annotation cost reduction and reinforcement learning from human feedback appear on 44 recent papers in the AI journals, among 71 papers on annotation and human feedback, but only 8 of them carry a worker, labour, wage, platform or ethics keyword. Millions of people label data and rate model outputs, often through platforms in low-income countries. What are they paid, and how does the pay shape the data?

## What prompted this

Since October 2025, 71 papers carry an annotation-cost, human-annotation, annotator, crowdwork, human-feedback, RLHF or data-labelling keyword (69 in AI journals): "annotation cost reduction" on 27, "reinforcement learning from human feedback" on 17, "human-in-the-loop annotation" on 7, "expert annotation" on 5, "data annotation" on 6. Only 8 of the 71 also carry a worker, labour, wage, crowdwork, gig, pay, employment, economics, incentive, platform, fairness or ethics keyword; "ghost workers" and "precarious employment" appear on one and two papers.

## Questions

- What do annotators and raters earn per hour on the main platforms by country and task, and how has pay changed as demand for preference ratings grew?
- Does pay level change label quality and the preferences that are encoded in models, for example toward speed over care?
- Where is annotation work moving, and do the new locations' labour laws cover it?

## Data that could help

Platform task postings and pay rates, worker surveys, and experiments that vary pay and measure label quality.

## Next step

Scrape posted pay rates for annotation and rating tasks on the main platforms over a year and relate them to task type and worker location.

## Related research activity

*The keywords this note follows, in the tracked finance, accounting, economics, management, operations research and AI journals as of 2026-10-26*

| Keyword | Papers, last 30 days | Rank | Papers per month, Oct 23 – Oct 26* | Papers, last 36 months |
|---|---:|---:|---|---:|
| annotation cost reduction | 0 | – | 0, 0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 4, 2, 1, 1, 3, 4, 0, 2, 2, 1, 3, 4, 3, 1, 2, 6, 5, 6, 2, 2, 0, 0, 4, 3, 0 | 70 |
| reinforcement learning from human feedback | 0 | – | 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 3, 1, 3, 1, 1, 1, 0, 6, 1, 2, 1, 0 | 24 |
| data annotation | 0 | – | 0, 0, 0, 0, 0, 0, 1, 2, 1, 2, 1, 0, 1, 0, 0, 3, 0, 0, 3, 1, 0, 0, 0, 0, 2, 0, 0, 1, 0, 0, 2, 0, 1, 0, 1, 0, 0 | 22 |

## Cite this page

Scholar4AI. (2026). The annotation labour behind AI. Journal Research Rankings. Retrieved October 23, 2026, from https://scholar4ai.com/ideas/annotation-labour-behind-ai

```bibtex
@misc{scholar4ai-ideas-annotation-labour-behind-ai,
  author = {{Scholar4AI}},
  title = {The annotation labour behind AI},
  year = {2026},
  howpublished = {\url{https://scholar4ai.com/ideas/annotation-labour-behind-ai}},
  urldate = {2026-10-23},
  note = {Journal Research Rankings; data from OpenAlex (CC0); licensed CC BY-NC 4.0}
}
```

License: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/).