Evaluating fraud detection in money, not F1 scores
Class imbalance is one of the largest topics in the AI journals, and fraud detection is where it matters most for finance. But only 10 of 174 recent fraud papers carry a cost, profit or loss keyword; most report accuracy or F1 scores. Would ranking fraud models by the money they save change which ones win?
What prompted this
Since October 2025, "class imbalance" appears on 471 papers (427 in AI journals), and in the AI journals "label imbalance" is rising (7 papers in July–September 2026 against about 1 expected). Of the 174 papers with a fraud keyword (58 in finance journals, 57 in AI, 25 in economics, 16 in accounting), only 10 carry a cost-sensitive, profit, business-value, loss or cost keyword.
Questions
- On public fraud datasets, how often is the model with the best F1 score not the one that saves the most money once losses and investigation costs are counted?
- Do the oversampling methods popular in AI papers still help when the score is money?
- How should the alert threshold be set when investigators can check only a fixed number of cases a day?
Data that could help
Public fraud datasets with transaction amounts, such as the credit card fraud, PaySim and IEEE-CIS data in the dataset ranking's list, and the models of published papers.
Next step
Re-evaluate five published fraud models on a public credit card dataset by the money they save under a fixed daily investigation budget.