TL;DR. A language model that invents a citation is not malfunctioning. It is doing exactly what it was scored to do. In September 2025 four researchers, three at OpenAI and one at Georgia Tech, published an argument that hallucination survives because of how models are graded: on a binary correct-or-incorrect scale, a model that guesses always beats an identical model that admits uncertainty. They surveyed ten leading evaluations and found nine use binary grading, and not one awards full credit for a calibrated “I do not know.” We took OpenAI’s own SimpleQA results and computed a column their paper does not report: the bluff rate, meaning the share of failures where the model answered wrongly instead of abstaining. It ranges from 20.5% to 99.0%, a 4.8x spread, and it is close to independent of accuracy. Then we checked Vectara’s hallucination leaderboard, updated 11 May 2026, for what happens when the facts are supplied in the prompt. Grounding helps a great deal, but it produced a result we did not expect: in four separate within-family comparisons, raising reasoning effort raised the hallucination rate, and the summaries got longer every single time. Below: the five verified mechanisms, which regime your task is in, and a detection test for each.
This is the mechanical layer under our work on judging AI output and trust calibration. Those pieces tell you when to trust. This one tells you why the failures happen where they do.
The one-sentence answer
Models hallucinate because guessing scores better than abstaining, and everything else is detail about when.
That framing comes from “Why Language Models Hallucinate,” posted to arXiv on 4 September 2025 by Adam Tauman Kalai and Ofir Nachum of OpenAI, Santosh S. Vempala of Georgia Tech, and Edwin Zhang of OpenAI. Their analogy is a student in an exam. Faced with a question you cannot answer, a blank scores zero and a guess scores zero or one. Under that rule the rational strategy is always to guess, and to guess specifically, because “September 30” and “sometime in autumn” are graded identically when both are wrong but only the first can be right.
Language models, the authors argue, live permanently in that exam. They are optimised against benchmarks that mirror standardised tests, and so they are always in test-taking mode.
The paper’s most concrete contribution is a survey of what the benchmarks actually do. The authors examined ten widely used evaluations and classified each on whether its primary metric is strict correct-or-incorrect accuracy, and whether an abstention can earn any credit at all.
| Benchmark | Scoring method | Binary grading | Credit for “I don’t know” |
|---|---|---|---|
| GPQA | Multiple-choice accuracy | Yes | None |
| MMLU-Pro | Multiple-choice accuracy | Yes | None |
| IFEval | Programmatic instruction verification | Yes | None |
| Omni-MATH | Equivalence grading | Yes | None |
| WildBench | Language-model-graded rubric | No | Partial |
| BBH | Multiple-choice / exact-match | Yes | None |
| MATH (Level 5 split) | Equivalence grading | Yes | None |
| MuSR | Multiple-choice accuracy | Yes | None |
| SWE-bench | Patch passes unit tests | Yes | None |
| HLE | Multiple-choice / equivalence grading | Yes | None |
Source: Kalai, Nachum, Vempala and Zhang, “Why Language Models Hallucinate,” arXiv preprint 2509.04664v1, September 2025, Table 2. Our tally of their rows: 9 of 10 use binary grading; 9 of 10 give no credit for abstention; 0 of 10 give full credit for a calibrated abstention. WildBench’s rubric offers only partial credit, and the authors note its 1-to-10 scale may still score an abstention below a confident but hallucinated answer.
That last line in the note is the whole problem in miniature. Even the one benchmark that does not use binary grading may still reward a confident falsehood over an honest “I don’t know.”
The authors’ proposed fix is not a new hallucination benchmark. It is to change the scoring of the benchmarks that already dominate leaderboards, by stating a confidence threshold in the instructions: answer only if you are more than t confident, since mistakes are penalised t/(1-t) points, correct answers earn 1 point, and “I don’t know” earns 0.
A note on that formula, for readers who check arithmetic. The preprint lists natural values as t = 0.5 (penalty 1), t = 0.75 (penalty 2) and t = 0.9 (penalty 9). Applying the paper’s own stated formula, t/(1-t), gives 1 for t = 0.5 and 9 for t = 0.9, both matching, but 3 for t = 0.75, not 2. The parenthetical for the middle value is inconsistent with the formula it accompanies. This is a small slip in an otherwise careful preprint and it changes nothing about the argument. We flag it because it is exactly the kind of derived number this article is telling you to recompute, and the lesson does not stop at papers from frontier labs. The observation applies to version 1 of the arXiv preprint dated 4 September 2025; later published versions may differ.
The bluff rate: an original column
Here is where the theory becomes measurable.
OpenAI released SimpleQA in November 2024, a benchmark of 4,326 short fact-seeking questions written to be hard for GPT-4o, each with a single indisputable answer. Its grading has three outcomes rather than two: correct, incorrect, and not attempted, where “not attempted” covers answers that do not fully give the reference answer, including outright “I don’t know.”
That third category is what makes the dataset useful here, because it separates two things that binary benchmarks fuse: not knowing, and pretending to know.
The published table reports correct, not attempted, and incorrect for eight models. It does not report the ratio that matters most for a working professional. If a model is going to fail you, what does that failure look like? Does it tell you it does not know, or does it hand you something false with a straight face?
That is a single division: incorrect divided by the sum of incorrect and not attempted. We computed it for every row.
| Model | Correct (%) | Not attempted (%) | Incorrect (%) | Bluff rate (%) |
|---|---|---|---|---|
| GPT-4o-mini | 8.6 | 0.9 | 90.5 | 99.0 |
| GPT-4o | 38.2 | 1.0 | 60.8 | 98.4 |
| OpenAI o1-preview | 42.7 | 9.2 | 48.1 | 83.9 |
| OpenAI o1-mini | 8.1 | 28.5 | 63.4 | 69.0 |
| Claude-3.5-sonnet (2024-06-20) | 28.9 | 35.0 | 36.1 | 50.8 |
| Claude-3-opus (2024-02-29) | 23.5 | 39.6 | 36.9 | 48.2 |
| Claude-3-haiku (2024-03-07) | 5.1 | 75.3 | 19.6 | 20.7 |
| Claude-3-sonnet (2024-02-29) | 5.7 | 75.0 | 19.3 | 20.5 |
CEOtudent editorial framework. The correct, not-attempted and incorrect columns are reported verbatim from OpenAI, “Measuring short-form factuality in large language models,” arXiv preprint 2411.04368, Table 3. The bluff rate is our calculation: incorrect / (incorrect + not attempted), that is, the share of non-correct responses in which the model asserted a wrong answer rather than abstaining. Every row of the source table sums to 100.0, which we verified. Models and dates are as evaluated in late 2024 and current systems will differ; the column is included to show the spread in abstention behaviour, not to rank vendors today.
Two readings, and the second is the important one.
The obvious reading is the spread. GPT-4o-mini, when it does not know, tells you anyway 99 times out of 100. Claude-3-sonnet, in the same situation, declines four times out of five. That is a 4.8x difference in how failure presents itself.
The reading that matters more: bluff rate is largely decoupled from accuracy. GPT-4o-mini answered 8.6% of questions correctly. Claude-3-haiku answered 5.1% correctly. A 3.5 percentage point difference in competence, and a 78 point difference in how they behave when out of their depth. These two models know almost equally little. One tells you so and one does not.
This is precisely what the hallucination paper predicts. Abstention behaviour is not a byproduct of capability. It is a separate, trained disposition, set by what the training and evaluation process rewarded. And because nine of ten leading benchmarks pay nothing for abstention, the incentive gradient points one way.
The practical consequence: you cannot infer trustworthiness from benchmark scores. A model at the top of a leaderboard may be the one most likely to hand you a fabricated citation, because leaderboard position and willingness to bluff are produced by the same pressure.
The second regime: what happens when you supply the facts
Everything above concerns closed-book recall, where the model answers from parameters alone. Most professional work is not like that. You paste a document, attach a report, point at a codebase. Does that fix it?
Vectara maintains a public hallucination leaderboard using its Hallucination Evaluation Model. The task is deliberately narrow: each model is given a document and asked to summarise it using only facts present in that document, then the summary is checked for statements the source does not support. The dataset holds more than 7,700 articles across news, technology, science, medicine, legal, sports, business and education, ranging from 50 to 24,000 words, is kept private to prevent overfitting, and models are called at temperature 0. The version below was last updated on 11 May 2026 and covers 106 models.
Across the full leaderboard, grounded hallucination rates run from 1.8% at the best to 24.2% at the worst.
Compare that to the closed-book numbers above, where GPT-4o answered 60.8% of SimpleQA questions incorrectly. These are different tasks, different models and different dates, so this is not a like-for-like model comparison and should not be read as one. What it is, is a comparison of two regimes, and the gap between them is roughly an order of magnitude. Putting the fact in the context window is the single largest intervention available to you, and it is available on every task, for free, right now.
But it does not take the rate to zero, and the shape of the residual error is where it gets interesting.
The finding we did not expect: reasoning effort makes grounded hallucination worse
While reading the leaderboard we noticed that several model families appear more than once at different reasoning settings. That creates natural controlled comparisons: same family, same benchmark run, same date, one variable changed.
There are five such pairs. Here is what they show.
| Family | Lower-effort setting | Rate | Words | Higher-effort setting | Rate | Words | Change in rate | Change in length |
|---|---|---|---|---|---|---|---|---|
| GPT-5.2 | low | 8.4% | 126.5 | high | 10.8% | 186.3 | +2.4 pts | +59.8 words |
| GPT-5.1 | low | 10.9% | 165.5 | high | 12.1% | 254.4 | +1.2 pts | +88.9 words |
| Grok-4.1-fast | non-reasoning | 17.8% | 87.5 | reasoning | 19.2% | 99.5 | +1.4 pts | +12.0 words |
| Grok-4-fast | non-reasoning | 19.7% | 141.9 | reasoning | 20.2% | 173.9 | +0.5 pts | +32.0 words |
| o4-mini | low | 18.6% | 130.9 | high | 18.6% | 127.7 | 0.0 pts | -3.2 words |
CEOtudent editorial framework. All rates and average summary lengths are reported verbatim from the Vectara hallucination leaderboard as updated 11 May 2026; the pairing, the change columns and the length reading are ours. Four of five within-family pairs show hallucination rate and summary length rising together with reasoning effort. The fifth, o4-mini, shows no meaningful movement in either, which is consistent with the same relationship rather than an exception to it.
Four out of four pairs that moved, moved the same way, and moved on both variables at once. The fifth pair did not move on either variable. There is no case in this data where more reasoning effort produced a lower grounded hallucination rate.
The length column suggests why, and the mechanism is unglamorous. Higher reasoning effort produces longer output. GPT-5.1 at high effort writes 254 words where the low setting writes 166, a 54% increase. Every additional sentence in a summary is another opportunity to assert something the source document does not support. The model is not reasoning itself into falsehood; it is writing more, and unsupported claims accumulate with volume.
The same pattern shows up across the whole OpenAI family on that single leaderboard run: gpt-5.4-nano, the smallest model listed, sits at 3.1%, while o3-pro, a reasoning flagship, sits at 23.3%. That is a 7.5x spread within one vendor’s lineup, running the wrong way relative to model capability.
Two cautions before anyone over-reads this. The benchmark measures one narrow task, faithful summarisation of a supplied document, and a reasoning model’s advantages on maths, code and multi-step analysis are simply not what is being scored here. And these are single leaderboard measurements without published confidence intervals, so the two smallest gaps, 0.5 and 1.2 points, should be treated as directional rather than decisive. The consistency across independent families is what carries the weight, not any individual row.
The actionable version: for extraction and summarisation, where the job is to stay inside a source document, reaching for the biggest reasoning model is not automatically the safe choice, and it may be the opposite. Ask for shorter output. It is the same lever.
The five mechanisms, and what each one looks like
The hallucination paper enumerates the statistical factors behind these errors. Here they are, mapped to what you can actually do about each. The mechanisms are the paper’s; the detection tests and the exposure column are ours.
| Mechanism | What causes it | What it looks like | Detection test | Highest-exposure tasks |
|---|---|---|---|---|
| Arbitrary facts | The fact has no pattern to generalise from and appeared rarely or never in training. The paper formalises this with a singleton rate: if a fraction of facts appeared exactly once, base models should hallucinate on at least that fraction | Confident, specific, plausible, wrong. Dates, names, citations, figures, page numbers | Ask the same question in three separate sessions. Unstable answers mean it is guessing. Stable answers do not prove correctness but instability is close to proof of error | Citations, biographical detail, statistics, obscure entities |
| Poor model fit | The architecture represents the task badly, independent of knowledge. The paper’s example is letter counting | Fails at something trivially easy while succeeding at something hard. Character counts, string manipulation, precise arithmetic | Give it a task you can verify in two seconds by hand. If it fails there, distrust the class of task, not the topic | Counting, tokenising, exact string work, arithmetic on long numbers |
| Computational hardness | The problem is intractable. No system, however capable, escapes complexity limits | A fluent answer to something that has no efficient solution | Ask whether a correct answer is computable at all at this scale. If not, the fluent answer is decoration | Cryptographic queries, large combinatorial optimisation, exhaustive search |
| Distribution shift | The prompt is unlike anything in training. Out-of-distribution inputs break classifiers, and models are no exception | Odd failures on trick questions, unusual phrasings, or genuinely novel situations | Rephrase into conventional form and re-ask. If the answer changes, the original phrasing was the problem | Novel scenarios, deliberately unusual framing, very recent events |
| Garbage in, garbage out | The training corpus contained the error. The model reproduces it faithfully | Confidently repeats a common misconception, an outdated figure, or a widely circulated misquote | Check the primary source, not a search result. Widely repeated numbers are the highest-risk category precisely because repetition is what put them in training | Popular statistics, folk science, contested history, superseded standards |
CEOtudent editorial framework. The five mechanisms and their technical basis are drawn from Kalai, Nachum, Vempala and Zhang, arXiv preprint 2509.04664v1, sections 3.3 and 3.4. The detection tests, exposure ratings and task mapping are our synthesis and are not findings of that paper.
The fifth row deserves emphasis because it is the one that catches careful people. A statistic repeated in a hundred articles is more likely to be reproduced by a model than a statistic reported once in the primary source, and repetition is not evidence. We have run into this directly: numbers that are ubiquitous in secondary coverage and simply absent from the underlying study.
The rule that follows is uncomfortable and correct. If a number matters, open the primary document and find the number in it. Not a summary of the document. The document.
A working protocol
Five steps, in the order that catches the most for the least effort.
Establish which regime you are in. Is the answer in the context window or in the model’s parameters? Closed-book recall is roughly an order of magnitude riskier. If you can convert a closed-book question into a grounded one by pasting the source, do that first. It is the highest-leverage move available.
Never accept a citation you have not opened. Titles, authors, years, journal names and page numbers are the canonical arbitrary-fact failure: no pattern to generalise, high plausibility when fabricated. Treat every unopened reference as unverified.
Re-ask, in a fresh session. Cheap, fast, and it exploits the mechanism directly. A model guessing at a singleton fact will produce different guesses. The hallucination paper opens with exactly this: asked for one author’s birthday across three attempts, a frontier model produced three different wrong dates.
Ask for shorter output on grounded tasks. The leaderboard pairs show length and unsupported claims rising together. Length is the lever you control.
Ask for the confidence, and give abstention permission. Say explicitly that “I don’t know” is an acceptable answer. This is the user-side version of the paper’s proposed fix. You are not going to change how benchmarks are scored, but you can change the scoring rule inside your own prompt.
What this does not mean
It does not mean the models are unreliable in general. Grounded rates of 2 to 7% for good models on faithful summarisation is a workable error rate for a first pass, provided you know it is not zero and you check the parts that matter.
It does not mean hallucination will be engineered away next year. The paper’s argument is structural: as long as the dominant evaluations pay nothing for calibrated uncertainty, models optimised for those evaluations will keep guessing. Fixing it requires changing the scoreboard, which is a coordination problem across the whole field rather than a modelling problem at one lab.
And it does not mean you should verify everything, which is just refusing to use the tool. It means you should know which of the five mechanisms your specific task is exposed to, and check there. Verification effort spent uniformly is verification effort wasted.
The competence being described here is not scepticism. It is calibration: knowing where the errors live so you can stop looking everywhere else. That is the same skill discussed in AI literacy versus AI fluency, applied to the failure surface rather than the capability surface.
Frequently asked questions
Do newer models hallucinate less?
Not reliably, and not on every task. On the grounded summarisation leaderboard updated in May 2026, the smallest model in OpenAI’s listed family scored 3.1% while a reasoning flagship in the same family scored 23.3%. Capability and faithfulness are different axes, and on this particular task they can point in opposite directions.
Does retrieval-augmented generation solve it?
It moves you into the better regime, which is a large gain, but the grounded rates in this article are measured with the source document already supplied. Every one of those numbers is a residual error that grounding did not remove. Retrieval also adds a failure mode of its own: retrieving the wrong document produces a confident answer faithfully grounded in the wrong source.
Why does the model sound so certain when it is wrong?
Because specificity was rewarded and hedging was not. Under binary grading, a vague answer and a wrong answer score identically at zero, while a specific guess has a chance of scoring one. Fluent confidence is the trained output of that incentive, not a signal about the answer.
Is asking the model whether it is sure any use?
Weakly. Stated confidence is systematically overstated. The SimpleQA authors measured this directly by asking models to state a confidence percentage and comparing it to their actual accuracy, and found the models consistently overstate. Repeated sampling across fresh sessions is a better signal than self-report, because it measures behaviour rather than a claim about behaviour.
Which single habit catches the most errors?
Opening primary sources for any number or citation that will affect a decision. It targets the two mechanisms responsible for most consequential professional errors, arbitrary facts and garbage in, garbage out, and it takes about a minute per item.
Sources
- Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, “Why Language Models Hallucinate,” arXiv preprint 2509.04664 version 1, September 2025. Table 2 on benchmark scoring; sections 3.3 and 3.4 on error mechanisms; section 4.2 on explicit confidence targets.
- OpenAI, “Measuring short-form factuality in large language models,” arXiv preprint 2411.04368, November 2024, introducing the SimpleQA benchmark. Table 3 on model performance; section 4 on calibration.
- Vectara, Hallucination Leaderboard, computed with the Hallucination Evaluation Model on a private corpus of over 7,700 documents, leaderboard version updated 11 May 2026.
- Vapnik and Chervonenkis, foundational work on the dimension of function families, cited by Kalai and colleagues as the basis for the arbitrary-facts analysis.
- I. J. Good, on the estimation of population frequencies of unseen species, the missing-mass estimator underlying the singleton-rate argument.
- Stanford Institute for Human-Centered Artificial Intelligence, Artificial Intelligence Index Report, on model evaluation practice and benchmark saturation.
This content was compiled with the support of AI following in-depth research, then written and prepared for publication by the CEOtudent editorial team.
This post is also available in:
