TL;DR. A citation is not a check. In a human audit of four generative search engines, only 51.5% of generated sentences were fully supported by their citations. In a 2025 study of medical answers, even GPT-4o with web search left 30.3% of its statements unsupported by its own sources; a 2025 preprint audit of search and deep research systems found unsupported-statement rates from 12.5% to 97.5%. Asking the model to re-check itself is weak: without outside feedback, GPT-3.5’s accuracy on one reasoning benchmark fell from 75.8% to 41.8%. What works is what professional fact-checkers do: split the answer into claims, open the source, find the exact sentence, and read laterally to learn who stands behind it. The CEO move: write a verification standard for anything that ships under your name, saying what must be traced, by whom and to what depth. The student move: practise claim decomposition and lateral reading, and keep a log of what you checked and what failed.
This guide belongs to the evaluation skill cluster. The red flags guide covers warning signs visible inside a single answer, the verification tax covers what checking costs, and why AI hallucinates covers the mechanics. This piece is the procedure you run before you rely on an answer.
Why do you need a procedure rather than a feeling?
Because the feeling points the wrong way. When Nelson Liu, Tianyi Zhang and Percy Liang at Stanford audited four generative search engines on 1,450 queries in early 2023, citation precision was inversely correlated with perceived utility: the responses that seemed most helpful were often those with more unsupported statements or inaccurate citations.
Trust also switches checking off. In two preregistered experiments with 5,387 participants in total, Folco Panizza and colleagues showed some participants a pop-up advising lateral reading and click restraint. In an exploratory analysis of the first experiment, only 6.7% of those participants used the techniques when a post came from a trusted source, against 20% when the source was unknown. An AI answer with tidy footnotes looks like a trusted source. A written procedure ties checking to the type of claim and the stakes, not to how the answer feels.
Does the citation actually support the claim?
Three studies have checked, statement by statement, whether AI answers are backed by the sources they cite. They differ in task (web questions, medical questions, debate and expertise queries), grading (human annotators; a GPT-4o judge that matched the consensus of three doctors 88.7% of the time, against 86.1% average agreement between the doctors; a GPT-5 judge validated against human raters), date (2023 to August 2025) and unit. Liu and colleagues required a statement’s own citations to support it fully; Kevin Wu, Eric Wu, James Zou and colleagues accepted any source in the response; DeepTRACE, a September 2025 preprint by Pranav Narayanan Venkit and colleagues, accepted any listed source. The table is descriptive, not a ranking.
Does the citation support the claim? Share of statements supported and not supported, by system
| Study (grading; period) | System, as labelled | Statements supported (%) | Statements not supported (%) | Citations that support their statement (%) |
|---|---|---|---|---|
| Liu, Zhang and Liang (human annotators; responses from February-March 2023) | Bing Chat | 58.7 | 41.3* | 89.5 |
| NeevaAI | 67.6 | 32.4* | 72.0 | |
| perplexity.ai | 68.7 | 31.3* | 72.7 | |
| YouChat | 11.1 | 88.9* | 63.6 | |
| Wu et al., Nature Communications (GPT-4o judge validated by doctors; 800 medical questions) | GPT-4o (RAG) | 69.7 | 30.3* | n/r |
| Gemini Ultra 1.0 (RAG) | 55.6 | 44.4* | n/r | |
| GPT-4o (API) | 47.3 | 52.7* | n/r | |
| Claude v2.1 (API) | 30.4 | 69.6* | n/r | |
| Mistral Medium (API) | 27.8 | 72.2* | n/r | |
| Mixtral-8x7b (API) | 15.8 | 84.2* | n/r | |
| Gemini Pro (API) | 18.6 | 81.4* | n/r | |
| DeepTRACE, Narayanan Venkit et al., preprint (GPT-5 judge; corpus as of 27 August 2025) | You.com (search) | 69.2* | 30.8 | 68.3 |
| BingChat (search) | 76.9* | 23.1 | 65.8 | |
| Perplexity (search) | 68.4* | 31.6 | 49.0 | |
| GPT 4.5 (search) | 53.0* | 47.0 | 39.8 | |
| GPT-5 Deep Research | 87.5* | 12.5 | 79.1 | |
| YouChat Deep Research | 25.4* | 74.6 | 72.3 | |
| GPT-5 Web Search | 41.1* | 58.9 | 31.4 | |
| Perplexity Deep Research | 2.5* | 97.5 | 58.0 | |
| Copilot Think Deeper | 9.8* | 90.2 | 62.1 | |
| Gemini Deep Research | 46.4* | 53.6 | 50.3 |
CEOtudent analysis of Liu, Zhang and Liang (2023), Wu et al. (2025, Supplementary Table 4) and Narayanan Venkit et al. (2025, Figure 2 and Table 1). Unmarked figures are as printed (Wu et al.’s proportions shown as percentages); * = 100 minus the printed share. n/r = not reported. The DeepTRACE text gives Gemini Deep Research’s citation accuracy as 40.3% but its Table 1 prints 50.3%; the table value is shown.
Four things stand out.
- Even the best system in each study left a large share unsupported: 31.3% in Liu’s audit, 30.3% in Wu’s, 12.5% in DeepTRACE. In 19 of the 21 rows at least a quarter of statements lacked support; the medians were 36.9%, 69.6% and 50.3%.
- Web access helps but does not settle it. In Wu’s study, models without web access produced working URLs only 41.2% to 68.7% of the time, and few broken links had archived versions, which the authors read as a sign they were invented. In DeepTRACE, GPT-5 Deep Research left 12.5% unsupported, GPT-5 Web Search 58.9%.
- Whole answers fare worse than single statements. In Wu’s study, 44.7% to 93.0% of answers held at least one unsupported statement. The Princeton ALCE benchmark (EMNLP 2023) found around 50% of its ChatGPT and GPT-4 baseline answers on the ELI5 dataset not fully supported by the cited passages.
- More sources do not mean more support. YouChat Deep Research listed 57.2 sources per answer, about 3.1 times GPT-5 Deep Research’s 18.3, and left 74.6% of statements unsupported. In the authors’ words, “more sources and longer answers do not translate into reliability.”
Treat these figures as an outside view: in none of the 21 rows is the base rate for “this cited statement is not supported” below one in eight (12.5%).
What do professional fact-checkers do that others do not?
The clearest evidence on how skilled people check unfamiliar information comes from Sam Wineburg and Sarah McGrew at Stanford, who watched 45 people think aloud while judging live websites: 10 PhD historians, 10 professional fact-checkers and 25 Stanford undergraduates.
How fact-checkers, historians and students evaluated unfamiliar websites
| Measure (task) | Fact-checkers (n = 10) | PhD historians (n = 10) | Undergraduates (n = 25) |
|---|---|---|---|
| Mean time to leave the American College of Pediatricians landing page (bullying task) | 32 s | 88 s (8 of 10 left) | 100 s (the 16 who left; 9 never left) |
| Mean quality of conclusion about that site (0-2) | 2 | 0.7 | 0.16 |
| Mean quality of conclusion about minimumwage.com before prompting (0-4) | 3.3 | 1.3 | 0.52 |
| Mean time to identify the site’s parent organisation | 51 s | 3 min 40 s | 5 min 18 s |
| Needed a prompt to find it | Found it without prompting | 6 of 10 | About four-fifths |
| Mean time to identify who funded a lawsuit (Vergara task) | 2 min 8 s | 2 min 1 s | 1 min 42 s |
Source: Wineburg and McGrew, Lateral Reading and the Nature of Expertise, Teachers College Record (pre-publication version dated 28 July 2018). Figures as printed.
The difference was method. Fact-checkers took bearings, read laterally (leaving a site after a quick scan to learn what others say about it) and practised click restraint (scanning search results before clicking). On the bullying task, 60% of students judged the College, a splinter group, more reliable than the American Academy of Pediatrics; every fact-checker chose the Academy. On the lawsuit task fact-checkers were slowest, yet “the most selective when it came to the sites they visited.” One confirmed a name by searching a news article with the command-F shortcut, the heart of Step 3 below.
Most people lack the skill, and it can be taught. In an assessment of 3,446 US high school students, 96% never learned that a site claiming to report climate facts had ties to the fossil fuel industry. In instructor-matched civics sections studied by Jessica Brodsky and colleagues (136 students taught lateral reading, 94 controls), the share who read laterally and reached a correct verdict on a fake news problem rose from 2.2% to 43.7%; controls ended at 1.1%. A district-wide field study by Wineburg and colleagues, with six 50-minute lessons for 271 students against 228 in regular classrooms, found the taught students “grew significantly” in judging credibility. A 2024 Nature Human Behaviour toolbox drawn from 81 scientific papers lists lateral reading among nine main intervention types against misinformation.
For AI work the translation is direct: an AI answer is an unfamiliar website with no About page. Reading it more carefully is vertical reading; checking it means leaving it.
Why not just ask the model “are you sure?”
Because without information from outside the conversation, a model has little to correct against. Jie Huang and colleagues at Google DeepMind and the University of Illinois tested “intrinsic” self-correction, where the model reviews its own answer with no external feedback (ICLR 2024).
Accuracy before and after intrinsic self-correction (%)
| Model | Benchmark | Standard prompting | After two rounds of self-correction | Change (points) |
|---|---|---|---|---|
| GPT-3.5 | GSM8K (maths) | 75.9 | 74.7 | -1.2* |
| GPT-3.5 | CommonSenseQA | 75.8 | 41.8 | -34.0* |
| GPT-4 | GSM8K (maths) | 95.5 | 89.0 | -6.5* |
| GPT-4 | CommonSenseQA | 82.0 | 80.0 | -2.0* |
Source: Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024, Table 3 (GSM8K and CommonSenseQA rows). Figures as printed; * change computed by CEOtudent.
Gains appeared only when the correction step was told whether the answer was right (GPT-3.5 on GSM8K rose from 75.9 to 84.3 with these “oracle” labels), information you lack in real use. The authors’ explanation is blunt: models “cannot properly judge the correctness of their reasoning”. For the separate finding that a plain “Are you sure?” often flips a correct answer, see the red flags guide.
Structured verification does better. In Chain-of-Verification (Shehzaad Dhuliawala and colleagues at Meta AI, Findings of ACL 2024), the model drafts an answer, plans verification questions, answers them separately and revises. On Wikidata list questions, precision more than doubled, from 0.17 to 0.36; on biographies, FactScore rose 28%, from 55.9 to 71.4. Two details matter for people too. Answering verification questions without the draft in view beat answering them alongside it (FactScore 63.7 against 60.8), since a checker who sees the draft may repeat its errors. And in the factored setting, yes/no checks did worse than open questions; in the authors’ examples, ChatGPT “tends to agree with facts in a yes/no question format whether they are right or wrong.”
The protocol below borrows all three lessons: check against something outside the draft, ask open questions rather than “is it true that…?”, and answer them without the draft steering you.
The 5-step verification protocol
CEOtudent editorial framework: the 5-step verification protocol
| Step | What to do | The question to ask | Evidence it rests on | Stop rule |
|---|---|---|---|---|
| 1. Decompose | Split the output into single, checkable claims and tag each by type. Mark opinion and framing separately. | “What exactly is asserted here, and could this one sentence be wrong on its own?” | Citation studies grade statement by statement; in Chain-of-Verification, short separate questions were answered more accurately than the original query | Every sentence is either on the claim list or marked as framing with no factual content |
| 2. Triage | Mark each claim’s stakes and pick a depth level. Flag claims that would change a decision if wrong. | “If this is wrong, who is affected, and can it be undone?” | Support rates vary from 2.5% to 87.5% of statements by system (table above), so no blanket trust level is safe | Level 1 claims may stay untraced if labelled unchecked; every critical claim goes to Step 3 |
| 3. Trace | Open the cited (or primary) source. Search it with Ctrl+F or command-F for the key term or number. Copy the supporting sentence. | “In which sentence, on which page, does the source say this?” | Only 51.5% of statements fully supported and 74.5% of citations supporting their statement (Liu et al.); fact-checkers locate exact text (Wineburg and McGrew) | Supported only if you can paste the sentence. Partial: narrow the claim to it. Nothing after two targeted searches: unsupported, do not use |
| 4. Read laterally | Leave the source. Find out who is behind it and whether an independent source, or the primary record, says the same. | “Who published this, what do others say about them, and does an independent source agree?” | Fact-checkers outperformed historians and students by leaving sites early; the skill is trainable (Brodsky et al.; Wineburg et al.) | Two independent sources agree, or you hold the primary record itself (statute, filing, dataset, original paper) |
| 5. Recompute and log | Redo every calculation from the source’s inputs; check year, unit, denominator and edition. Log the verdict. | “Does each number survive recalculation, and could someone else repeat my check from the log?” | Self-review without outside information lowered accuracy (Huang et al.); independent verification raised it (Dhuliawala et al.) | Each claim has a logged verdict: keep, narrow, cut or escalate |
One practical note: in Step 1, split compound sentences. “Sales rose sharply after the new law” holds three claims: that sales rose, by how much, and why.
How deep should you go?
Not every output deserves all five steps; the stakes triage sets the depth. It complements the time-budget tiers in the verification tax and the checkpoint design in human-in-the-loop by design.
CEOtudent editorial framework: stakes triage
| Level | Typical outputs | Steps to run | Who checks |
|---|---|---|---|
| 1. Private and reversible | Notes to self, brainstorms, first drafts, search starting points | Steps 1-2 on the whole output; Steps 3 and 5 for any number, name or quote you will reuse | You |
| 2. Shared or decision-relevant | Internal memos, analyses that inform a team decision, code merged after review | All five steps for every critical claim; Step 4 for any source you do not already know | You, with the log attached |
| 3. Under your name, irreversible or regulated | Client deliverables, published articles, legal, medical, financial or safety content | All five steps for every factual claim; two independent sources or the primary record for each critical claim | You, plus a second reader who re-runs a sample of the log |
Move up a level when in doubt. Moving down needs a reason you could defend if the output turned out to be wrong.
Which check fits which claim?
Different claims fail in different ways, so the fastest primary check differs too.
CEOtudent editorial framework: claim-type routing matrix
| Claim type | Fastest primary check | Typical failure mode |
|---|---|---|
| Number or statistic | The original publisher’s table or dataset; match year, edition, unit and denominator | Real number from the wrong year or base; a derived figure presented as printed |
| Direct quote | Search the exact phrase in the original text or transcript; check speaker and context | Paraphrase presented as a quote; right words, wrong speaker or missing context |
| Citation or reference | Look up the exact title or DOI on the publisher’s site, then run Step 3 on it | Reference that does not exist; real reference that does not say what it is cited for |
| Date or event | The dated primary record: official announcement, filing, publication page | Off by a year or a version; two events merged into one |
| Law or regulation | The current consolidated text on the official register, with amendment dates | Superseded provision or deadline quoted as current |
| Code behaviour | Run it on a test with a known answer; read the documentation for the installed version | Code that runs but does the wrong thing; a function or option that does not exist |
| Causal claim | The study’s design and effect size in the paper itself | Correlation stated as cause; one study generalised to everyone |
| “Current” or “latest” status | The publisher’s own latest-edition page and its date | Stale edition presented as the latest; search snippets lagging the live page |
The mechanics guide explains why arbitrary facts such as dates, titles and figures are where models most often guess.
A worked example: one sentence through five steps
The sentence below is an illustrative assumption written in the style of an AI summary, not output from any real system. “A Stanford study found that professional fact-checkers were faster than historians and students at every task, leaving unreliable websites four times faster.”
- Decompose. Three claims: (a) a Stanford study compared these groups; (b) fact-checkers were faster at every task; (c) they left sites four times faster.
- Triage. Destined for a client presentation, so Level 3.
- Trace. Claim (a) holds: Wineburg and McGrew studied 10 fact-checkers, 10 historians and 25 students. Claim (b) fails: on the lawsuit task students were fastest (1 min 42 s against 2 min 8 s). Claim (c) is partial: on the bullying task fact-checkers left the landing page in 32 seconds, historians in 88.
- Read laterally. The paper was accepted by Teachers College Record, an academic journal; later classroom studies support the method, not the speed claim.
- Recompute and log. 88 / 32 = 2.75, which the paper calls “almost three times”. On the minimum wage task, 3 min 40 s against 51 s is 4.3 times, which the paper’s “nearly four times” undersells. Verdict: narrow. Usable version: “In one Stanford study, fact-checkers left an unfamiliar site in a mean 32 seconds, against 88 for historians, but were not fastest on every task.”
The verification log: a template you can copy
Keep one row per checked claim. The log turns checking into a record a colleague can audit, and feeds the error review in learning from AI mistakes.
| # | Claim (one sentence) | Type | Level | Source opened (title, date, page) | Exact supporting sentence found? | Independent second source | Number recomputed | Verdict | Checked by, date |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Fact-checkers left sites four times faster | Number | 3 | Wineburg and McGrew, 2018 pre-publication, bullying and minimum wage tasks | Partial: 32 s against 88 s; 51 s against 3 min 40 s | Not needed: primary paper | Yes: 2.75 and 4.3 | Narrow | Initials, date |
One-page checklist before anything leaves your hands
- Every factual sentence is on the claim list, or marked as framing.
- Every critical claim has a pasted supporting sentence from the source itself.
- Every reference was found on the publisher’s site, not only in the AI answer.
- Every unfamiliar source was read laterally: who runs it, who funds it.
- Every number was recomputed from the source’s inputs, with year and unit checked.
- No claim rests only on the model’s own re-check.
- Level 3 work has a second reader’s initials on a sample of rows.
What does this mean for the CEO and the student?
The CEO move: write the standard down. Anything that ships under your name needs a one-page verification standard: which claim types must be traced (at minimum numbers, quotes, references and legal or medical statements in external work), who traces them, at which level, and where the log lives. Add one rule: no source sentence, no claim. Trust calibration decides when to rely on a recommendation; the standard decides what must be proven first.
The student move: train the two skills the protocol depends on. Claim decomposition is a drill: split any AI paragraph into single claims until none can be split further. Lateral reading is a habit: before reading a source closely, find out who is behind it. Note your confidence in each claim before checking and compare it with the verdict, as confidence calibration describes. Over time the log shows which claim types fool you most, the practical core of the discernment gap.
What this does not mean
It does not mean AI answers are usually wrong: a missing source is a different problem from a false claim. It does not mean the systems in the table rank in that order today; they were tested in 2023 to 2025. The lateral reading evidence comes mainly from students and websites, not from AI output, so its transfer to AI work is an inference. And verification does not make output true; it makes your confidence match the evidence you hold.
Frequently asked questions
How can you tell if an AI answer is true?
Only by tracing its critical claims to a primary source and finding the sentence that supports each one. Fluency, confidence and footnotes are not evidence; in one audit, the most helpful-seeming answers were often the least supported.
Can the AI check its own answer?
Use it to list claims and draft verification questions, not as the final check. Without outside information, self-correction lowered accuracy in the Huang et al. tests; structured, independent questioning improved it.
Are AI answers with citations more reliable?
Citations make checking possible, not unnecessary. Across the three studies above, the share of statements without support ranged from 12.5% to 97.5% depending on the system.
What is lateral reading?
Leaving a source to learn what independent sources say about it, instead of judging it from its own pages. In the Stanford study, fact-checkers left an unfamiliar page in about half a minute on average; classroom studies show it can be taught in a handful of lessons.
Is a second AI model an independent check?
Not on its own. It can flag inconsistencies, but the studies above show that even models with web search often cite sources that do not support them. The independent check is the primary source.
Sources
- Liu, N. F., Zhang, T. and Liang, P. (2023). Evaluating Verifiability in Generative Search Engines. Findings of EMNLP 2023. arXiv 2304.09848, version 2.
- Wu, K., Wu, E., Zou, J. and colleagues (2025). An automated framework for assessing how well LLMs cite relevant medical references. Nature Communications 16, 3615, and its Supplementary Information.
- Narayanan Venkit, P., Laban, P., Zhou, Y. and colleagues (2025). DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence. arXiv 2509.04499, version 1 (preprint by authors at Salesforce AI Research and Microsoft Research; not peer reviewed).
- Gao, T., Yen, H., Yu, J. and Chen, D. (2023). Enabling Large Language Models to Generate Text with Citations. EMNLP 2023. arXiv 2305.14627.
- Huang, J., Chen, X., Mishra, S. and colleagues (2024). Large Language Models Cannot Self-Correct Reasoning Yet. ICLR 2024. arXiv 2310.01798, version 2.
- Dhuliawala, S., Komeili, M., Xu, J. and colleagues (2024). Chain-of-Verification Reduces Hallucination in Large Language Models. Findings of ACL 2024, pages 3563-3578. arXiv 2309.11495.
- Wineburg, S. and McGrew, S. Lateral Reading and the Nature of Expertise: Reading Less and Learning More When Evaluating Digital Information. Teachers College Record. Pre-publication version dated 28 July 2018, Stanford Digital Repository.
- Breakstone, J., Smith, M., Wineburg, S. and colleagues (2021). Students’ Civic Online Reasoning: A National Portrait. Educational Researcher. Pre-publication version, Stanford Digital Repository.
- Brodsky, J. E., Brooks, P. J., Scimeca, D. and colleagues (2021). Improving college students’ fact-checking strategies through lateral reading instruction in a general education civics course. Cognitive Research: Principles and Implications 6, 23.
- Wineburg, S., Breakstone, J., McGrew, S. and colleagues (2022). Lateral Reading on the Open Internet: A District-Wide Field Study in High School Government Classes. Journal of Educational Psychology 114, 893-909 (abstract only, ERIC record EJ1372738).
- Panizza, F., Ronzani, P., Martini, C. and colleagues (2022). Lateral reading and monetary incentives to spot disinformation about science. Scientific Reports 12, 5678.
- Kozyreva, A. and colleagues (2024). Toolbox of individual-level interventions against online misinformation. Nature Human Behaviour, published 13 May 2024 (abstract only).
Note on methods. “Does the citation support the claim?” is a CEOtudent analysis of sources 1-3; asterisked cells, medians and ranges are computed from the printed figures, as is the change column in the self-correction table. The 5-step protocol, stakes triage, routing matrix and verification log are CEOtudent editorial frameworks; the worked example uses an illustrative sentence. We deliberately did not use the widely repeated 2025 Tow Center for Digital Journalism figures on AI search citations, because we did not open and check that report. EBU and BBC newsroom audit, court hallucination, fabricated reference, FActScore, SelfCheckGPT and semantic entropy figures appear in the linked cluster posts. This article is general information, not legal, medical or financial advice.
This content was compiled with the support of AI following in-depth research, then written and prepared for publication by the CEOtudent editorial team.
This post is also available in:
