---
title: "When the judge is a model (research idea)"
description: "Evaluation metrics appear on 56 recent papers and LLM-as-a-judge on 16, all in AI journals, with 25 evaluation papers in the last quarter in line with earlier quarters. Models now grade other models' answers, and firms use them to score customer conversations and reports. How reliable and how biased are the model judges, and how far do the biases propagate?"
url: https://scholar4ai.com/ideas/llm-as-a-judge-reliability
markdown_url: https://scholar4ai.com/ideas/llm-as-a-judge-reliability.md
license: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/)
share_text: "When the judge is a model (research idea): Evaluation metrics appear on 56 recent papers and LLM-as-a-judge on 16, all in AI journals, with 25 evaluation papers in the last quarter in line with earlier quarters."
date: "2026-10-23"
updated: "2026-10-23"
status: "seed"
tags: ["ai", "methods"]
keywords: ["LLM-as-a-judge", "evaluation metrics", "LLM evaluation"]
---

# When the judge is a model

Evaluation metrics appear on 56 recent papers and LLM-as-a-judge on 16, all in AI journals, with 25 evaluation papers in the last quarter in line with earlier quarters. Models now grade other models' answers, and firms use them to score customer conversations and reports. How reliable and how biased are the model judges, and how far do the biases propagate?

## What prompted this

Since October 2025, 106 papers carry an LLM-as-a-judge, automatic-evaluation, evaluation-metric or human-evaluation keyword (97 in AI journals): "evaluation metrics" on 56, "LLM-as-a-judge" on 16, "LLM evaluation" on 9, "benchmark evaluation" on 5, "human evaluation" on 3. The quarter to September 2026 had 25 such papers against about 26 per quarter earlier.

## Questions

- On tasks with expert ground truth in finance, accounting and law, how well do model judges agree with experts, and do they favour longer, more confident or more similar-sounding answers?
- When a model is trained on rewards from a model judge, do the judge's biases compound in the trained model?
- In business uses (scoring customer service calls, grading analyst notes), do the scores change behaviour in the direction the firm wants?

## Data that could help

Expert-graded answer sets in professional domains, judge outputs from several models, and training runs with model-judged rewards.

## Next step

Compare model-judge scores with expert grades on 300 answers to accounting and legal questions and measure length and self-preference biases.

## Related research activity

*The keywords this note follows, in the tracked finance, accounting, economics, management, operations research and AI journals as of 2026-10-26*

| Keyword | Papers, last 30 days | Rank | Papers per month, Oct 23 – Oct 26* | Papers, last 36 months |
|---|---:|---:|---|---:|
| LLM-as-a-judge | 0 | – | 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 2, 1, 1, 0, 2, 2, 2, 0, 3, 2, 0 | 18 |
| evaluation metrics | 4 | #525 | 4, 2, 2, 6, 5, 5, 1, 4, 0, 6, 3, 4, 5, 3, 10, 7, 5, 7, 8, 5, 5, 11, 6, 3, 3, 9, 5, 8, 1, 8, 6, 4, 9, 1, 6, 4, 3 | 181 |
| LLM evaluation | 2 | #1303 | 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 2, 1, 0, 0, 1, 1, 1 | 16 |

## Cite this page

Scholar4AI. (2026). When the judge is a model. Journal Research Rankings. Retrieved October 23, 2026, from https://scholar4ai.com/ideas/llm-as-a-judge-reliability

```bibtex
@misc{scholar4ai-ideas-llm-as-a-judge-reliability,
  author = {{Scholar4AI}},
  title = {When the judge is a model},
  year = {2026},
  howpublished = {\url{https://scholar4ai.com/ideas/llm-as-a-judge-reliability}},
  urldate = {2026-10-23},
  note = {Journal Research Rankings; data from OpenAlex (CC0); licensed CC BY-NC 4.0}
}
```

License: CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/).