Do multi-criteria rankings agree?
Supplier selection is one of the fastest-growing keywords in the operations research journals, and much of it relies on multi-criteria methods such as AHP, TOPSIS and the best-worst method. These methods can rank the same options differently. How often would a paper's conclusion change with the method, and what should a robustness check look like?
What prompted this#
In the operations research journals, "supplier selection" had 11 papers in July–September 2026 against about 4 expected (see the operations research dashboard). Since October 2025, 227 papers carry a multi-criteria keyword such as TOPSIS, AHP or the best-worst method (112 in AI journals, 74 in operations research). Only 15 of them carry a rank-reversal, sensitivity-analysis, robustness or comparative-analysis keyword. The formulas of these methods are on the equation ranking's list.
Questions#
- In published case studies, how often does the top choice change when the weighting method, the normalization or the aggregation changes?
- How sensitive are the rankings to small changes in expert judgments?
- Would reporting the share of method variants that agree on the winner be a useful standard?
Possible approach#
Re-run the decision matrices reported in recent papers through a grid of method variants, and record how often the top choice and the full ranking agree.
Next step#
Collect the decision matrices of 20 recent supplier-selection papers and re-rank each one with TOPSIS, VIKOR and the best-worst method.