Why Language Models Hallucinate – A Statistical + Incentive Story

The Casino Always Wins

We are awash in a sea of fluent nonsense. For the past few years, anyone building with LLMs has confronted the same unsettling truth: the models are masters of confident error. They bluff with the poise of a seasoned poker player, and we call this behavior “hallucination,” a term that mystifies what is, in reality, a predictable outcome.

The paper Why Language Models Hallucinate cuts through the fog of ad-hoc explanations with a clean, almost brutal, theoretical blade. It presents a unified theory, connecting the statistical pressures of pretraining with the perverse incentives of evaluation.

The framework splits the problem into two acts:

The Inevitable Sin of Pretraining. Here, the base model learns a probability distribution over all language. The paper shows that even with a perfect, error-free corpus, a well-calibrated model is statistically compelled to generate some invalid outputs. Some error is baked in from the start.
The Corrupting Influence of Evaluation. Next, we fine-tune these models against benchmarks that score correctness as a binary choice. In a world of 0s and 1s, saying “I don’t know” is a guaranteed zero. Guessing, however, offers a non-zero chance of a payoff. So, we systematically train our models to be gamblers, not scholars.

What emerges is a rare thing: a single, coherent story that explains not only where hallucinations are born, but why we can’t seem to kill them.

The Core Idea: Generation as Classification

The paper’s central move is an elegant reduction. It reframes the messy problem of generation into a clean binary classification task called Is-It-Valid (IIV): for any given output, is it valid (+) or an error (-)? From here, the authors show that any language model inherently acts as an IIV classifier by setting a probability threshold. This forges a hard, quantitative link between the model’s ability to classify and its tendency to hallucinate.

The Foundational Bound

The math is unforgiving. For a set of plausible strings $X$ , partitioned into valid strings $V$ and errors $E$ , a well-calibrated base model $\hat p$ will have a generative error rate bound by:

$\varepsilon_{\textrm{gen}} \;\ge\; 2 \varepsilon_{\textrm{IIV}}\; -\; \frac{|V|}{|E|}\; -\; \delta$

Here, $\varepsilon_{\textrm{IIV}}$ is the model’s error rate on the IIV classification task, and $\delta$ is a small term for calibration error. The logic is inescapable: if your system can’t reliably tell truth from fiction at a probabilistic level, it has no hope of generating only truth.

With Prompts

When we introduce prompts $c$ and responses $r$ , the bound tightens:

$\varepsilon_{\textrm{gen}} \;\ge\; 2 \varepsilon_{\textrm{IIV}}\; -\; \frac{\max_{c} |V_{c}|}{\min_{c} |E_{c}|}\; -\; \delta.$

Suddenly, two useful levers appear:

$\max_{c} |V_{c}|$ – The number of ways to be “right.” For many factual queries, this is just 1.
$\min_{c} |E_{c}|$ – The number of ways to be plausibly “wrong.” For arbitrary facts, this can be enormous.

The Illusion of $\delta$

One might hope that $\delta$ , the calibration error, is large enough to offer an escape. It isn’t. Because pretraining is a game of minimizing cross-entropy, the base model is already incentivized to be near a local optimum where $\delta$ is negligible. In other words, base models arrive on our doorstep already well-calibrated. They are honest, probabilistic machines before we teach them to be otherwise.

Arbitrary Facts and the Brutality of Singletons

The theory is most punishing in the regime of arbitrary facts – domains where there is no discernible pattern, like a person’s birthday. Here, the model can’t reason; it can only memorize. The paper proves that the hallucination rate is brutally tied to the singleton rate ( $s_{r}$ ), the fraction of facts that appear just once in the training data:

$\varepsilon_{\textrm{gen}} \;\gtrsim\; s_{r}\; -\; \frac{2}{|E_{c}|}\; -\; O\left(\frac{\log N}{\sqrt{N}}\right)\; -\; \delta,$

where $|E_{c}|$ is the number of plausible wrong answers (e.g., ~364 for a birthday) and $N$ is the number of training examples.

The takeaway is stark and matches the grim reality of working with long-tail knowledge: If 20% of the birthdays in your corpus are singletons, a perfectly calibrated base model is mathematically compelled to hallucinate on at least ~20% of birthday-related queries. It’s not a bug; it’s a direct consequence of the data’s structure.

Poor Models vs. Broken Incentives

Not all hallucinations are born from singletons. The usual suspects still play a role:

Poor Models. An insufficiently powerful model class – a trigram model trying to understand long-range grammar, for instance – will fail. Its classification error remains high, and so does its generative error.
Garbage In, Garbage Out. Errors in the pretraining data will, of course, be parroted back. This is the least surprising failure mode, though the paper’s bounds hold even with a perfectly clean dataset.

But the most damning insight comes from looking at our post-training rituals.

Why We Train Our Models to Lie

Most of our prized benchmarks – MMLU-Pro, GPQA, SWE-bench – operate on a simple, binary logic: you are right or you are wrong. In this unforgiving world, abstaining – admitting “I don’t know” – is a guaranteed zero. A wild guess, however, has a positive expected value. We have built an entire ecosystem that rewards confident guessing.

Observation: With binary graders, the optimal strategy is almost never to abstain. We are systematically training models to bluff.

This explains a deeply counterintuitive phenomenon: post-training often increases overconfidence, systematically destroying the base model’s more honest, well-calibrated nature. We are actively training the humility out of our machines.

The Fix: Change the Game

The most actionable idea is to change the rules of the game. We must embed explicit confidence targets into our prompts and, more importantly, our scoring functions. The authors propose appending guidance like this:

“Answer only if you are more than t confident. A wrong answer loses t/(1−t) points; a correct answer gains 1; IDK gets 0.”

At $t=0.5$ (50% confidence), the penalty for being wrong is 1 point.
At $t=0.75$ , the penalty is 3 points.
At $t=0.9$ , the penalty is 9 points.

This simple change converts the problem into one of behavioral calibration: the model must learn to output an answer if and only if its internal probability of being correct exceeds the target $t$ . Critically, this needs to be integrated into the mainstream leaderboards that the entire field optimizes against. This is how you realign the incentives of an entire ecosystem.

A Model of the Madness

A Builder’s Playbook

Demand Thresholded Answering. In system prompts, explicitly define a confidence target ( $t$ ) and a scoring rule. Train the model to this rule. In applications where failure is costly, the default behavior must be to abstain and trigger verification.
Audit Behavioral Calibration. Measure your model’s performance at various values of $t$ (e.g., 0.5, 0.75, 0.9). Track its abstention rate, the accuracy on questions it does answer, and the net score. Do this for every domain.
Divide and Conquer. Recognize different failure regimes and treat them accordingly.
- Arbitrary facts: Mitigate with retrieval (RAG), force abstention, or require user confirmation.
- Systematic tasks: Improve the model architecture or give it better tools (code interpreters, calculators, structured parsers).
Re-Score Your Evals. Apply these penalty-based scoring rules to your internal test harnesses. Stop rewarding teams for optimizing pass-rate at the expense of honesty.
Design for Uncertainty. Teach models a vocabulary of uncertainty (hedging, asking for clarification), not just a binary “IDK.” Surface these states clearly in the user interface.

The Math, In Practice: Estimating the Singleton Rate

This Python snippet computes the singleton rate from your data, giving a conservative lower bound on hallucinations for an arbitrary-facts task. Use it on a corpus of (prompt, answer) pairs, filtered to include only instances where the model provided a concrete answer.

from collections import Counter

# facts: list of prompts that have exactly one correct canonical form in your task
facts = [
    # e.g., "birthday:person123", "birthday:person456", ...
]

N = len(facts)
counts = Counter(facts)
singletons = sum(1 for k,v in counts.items() if v == 1)
# singleton rate among answered prompts
s_r = singletons / max(N, 1)

# Suppose the task format admits 364 plausible wrong answers per prompt (|E_c|)
E_c = 364
# A very loose finite‑sample cushion; tune for your N
finite_sample = 6 * ( (N>0) and (__import__('math').log(max(3*N,1)) / (N**0.5)) or 0 )

lower_bound = max(0.0, s_r - 2/E_c - finite_sample)
print({"singleton_rate": s_r, "conservative_hallucination_lb": lower_bound})

Interpretation: If singleton_rate is 0.18, your calibrated base model will likely hallucinate ≥18% of such queries unless you add retrieval, abstentions, or other mitigations.

Scoring Sanity: A Quick Function for Confidence Evals

Use this to grade a model’s behavior under a target threshold $t$ .

def score(decision, is_correct, t):
    """decision ∈ {"answer", "idk"}; is_correct ∈ {True, False}"""
    if decision == "idk":
        return 0.0
    return 1.0 if is_correct else -t/(1.0 - t)

Run this across a test set at $t\in\{0.5, 0.75, 0.9\}$ to audit your model’s calibration and inform which thresholds to use in production.

How Today’s Benchmarks Incentivize Guessing

Benchmark	Primary grading	IDK credit
MMLU‑Pro	Multiple‑choice accuracy	None
GPQA	Multiple‑choice accuracy	None
BBH	Multiple‑choice / exact‑match	None
MATH (L5)	Equivalence to ground truth	None
Omni‑MATH	Equivalence grading	None
SWE‑bench	Unit‑test pass/fail	None
IFEval	Programmatic rubric (binary sub‑scores)	None (in practice)
WildBench	LM‑graded 1‑10 rubric	Partial (IDK often < “fair” with minor errors)

Recommendation: These tasks are valuable. We should keep them, but augment them with confidence-targeted variants. Report both the conventional accuracy and the thresholded score.

Intellectual Lineage

This isn’t an idea born in a vacuum. It stands on the shoulders of giants:

Good–Turing “missing mass”. The classic result explaining why singletons are powerful predictors of unseen events. See I. J. Good (1953).
Calibrated models must hallucinate. An earlier paper by Kalai & Vempala (STOC’24) formalized the link between singletons and errors for arbitrary facts.
Evaluation ecosystems. HELM, WildBench, SWE-bench, and others are the perfect venues to inject these new, better-aligned scoring incentives.

High-authority resources to explore:

Good–Turing estimator (Biometrika, 1953): https://doi.org/10.1093/biomet/40.3-4.237
HELM Capabilities: https://crfm.stanford.edu/helm/capabilities/v1.5.0/
WildBench: https://openreview.net/forum?id=MKEHCx25xp
SWE‑bench: https://proceedings.iclr.cc/paper/2024/hash/edac78c3e300629acfe6cbe9ca88fb84
Humanity’s Last Exam: https://agi.safe.ai/
OpenAI explainer: https://openai.com/index/why-language-models-hallucinate/

Final Thoughts

So we arrive at a cleaner, colder truth. Hallucinations are not a mysterious ghost in the machine, nor are they purely a flaw in our architectures or datasets. They are the predictable, logical consequence of two forces: (i) the statistical realities of learning from sparse data, and (ii) the profoundly misaligned incentives in how we grade our creations. If we want systems we can trust, we must build systems that understand the value of saying “I don’t know.” That means we have to stop rewarding them for guessing.

Posted in AI / ML, LLM Intermediate, LLM Research by Rakshit Kalra