Evaluating large language models for accuracy incentivizes hallucinations

TL;DR

An incentive-based theoretical analysis showing that accuracy-based evaluations reward guessing in LLMs, leading to hallucinations.

Problem

LLMs still produce confident falsehoods (hallucinations), and existing approaches fail to fully explain the cause.

Approach

Using learning theory, we prove that next-word prediction training creates unavoidable errors for facts lacking repeated support, and that accuracy metrics reward guessing. We propose 'open rubric' evaluations to incentivize models to admit uncertainty.

Results & Contribution

By reframing hallucination as an incentive problem, we propose a new evaluation methodology that offers a practical path toward more reliable LLMs.