AI Judges Show Bias, But Align With Human Judgment in Risk Assessment

The increasing reliance on Large Language Models (LLMs) as evaluators in sensitive domains like financial risk assessment is raising critical questions about their reliability and inherent biases. A new structured multi-evaluator framework, combining a five-criterion rubric with Monte-Carlo scoring, has been developed to rigorously assess LLM reasoning in Merchant Category Code (MCC)-based merchant risk assessment. This framework, detailed in a recent arXiv preprint (arXiv:2602.05110v1), employs frontier LLMs to both generate and cross-evaluate risk rationales under distinct attributed and anonymized conditions.

Unpacking LLM Biases in Financial Risk

The research highlights significant heterogeneity among leading LLMs. Specifically, GPT-5.1 and Claude 4.5 Sonnet exhibit a negative self-evaluation bias, scoring an average of -0.33 and -0.31 respectively. Conversely, Gemini-2.5 Pro and Grok 4 display a positive bias, with scores of +0.77 and +0.71. Intriguingly, anonymizing the evaluation process reduced this bias by approximately 25.8 percent, suggesting that self-perception and an awareness of being judged play a role in LLM responses.

When evaluated by 26 human experts from the payments industry, the LLM judges, on average, assigned scores that were 0.46 points higher than human consensus. However, the study found that the negative bias observed in GPT-5.1 and Claude 4.5 Sonnet actually indicated a closer alignment with human judgment. This nuanced finding suggests that models exhibiting a more critical self-assessment might be more reliable when mirroring human decision-making patterns. The framework's validation against real-world payment network data confirmed that four of the tested models showed statistically significant alignment with ground truth, with Spearman correlation coefficients ranging from 0.56 to 0.77. This provides strong evidence that the developed framework is capable of capturing genuine quality in LLM-generated risk assessments.

The Imperative for Bias-Aware Protocols

These findings underscore a crucial point: while LLMs are demonstrating impressive capabilities in complex reasoning tasks, their deployment in high-stakes environments like financial risk assessment necessitates careful consideration of their inherent biases. The framework presented offers a replicable method for evaluating LLM-as-a-judge systems, paving the way for more trustworthy AI applications. The research implicitly calls for the development and implementation of bias-aware protocols, especially within operational financial settings, to ensure fairness, accuracy, and accountability.

It's tempting to see the higher scores from models like Gemini and Grok as a sign of superiority, but the alignment with human experts suggests a different story. The GPT and Claude models, with their more critical output, appear to be more accurately reflecting the cautious nature of human risk assessment. This highlights the ongoing challenge: building AI that not only performs well but also behaves in a manner that is predictable and interpretable, especially when human livelihoods and financial stability are on the line. The research provides a much-needed empirical foundation for understanding and mitigating LLM biases in a critical application domain.