When the judge is a model
Evaluation metrics appear on 56 recent papers and LLM-as-a-judge on 16, all in AI journals, with 25 evaluation papers in the last quarter in line with earlier quarters. Models now grade other models' answers, and firms use them to score customer conversations and reports. How reliable and how biased are the model judges, and how far do the biases propagate?
What prompted this#
Since October 2025, 106 papers carry an LLM-as-a-judge, automatic-evaluation, evaluation-metric or human-evaluation keyword (97 in AI journals): "evaluation metrics" on 56, "LLM-as-a-judge" on 16, "LLM evaluation" on 9, "benchmark evaluation" on 5, "human evaluation" on 3. The quarter to September 2026 had 25 such papers against about 26 per quarter earlier.
Questions#
- On tasks with expert ground truth in finance, accounting and law, how well do model judges agree with experts, and do they favour longer, more confident or more similar-sounding answers?
- When a model is trained on rewards from a model judge, do the judge's biases compound in the trained model?
- In business uses (scoring customer service calls, grading analyst notes), do the scores change behaviour in the direction the firm wants?
Data that could help#
Expert-graded answer sets in professional domains, judge outputs from several models, and training runs with model-judged rewards.
Next step#
Compare model-judge scores with expert grades on 300 answers to accounting and legal questions and measure length and self-preference biases.