TL;DR. In the OECD Survey of Adult Skills (2023 data, published December 2024), 25% of adults across OECD countries scored at or below Level 1 in numeracy and 14% reached Levels 4 or 5. In the United States the low-numeracy share rose from 29% in 2017 to 34% in 2023. Meanwhile AI tools and vendor decks produce more confident percentages than ever. Non-analysts need about ten ideas, each with one question attached, that stop a true number from misleading them. Format matters as much as the reader: when 160 gynecologists saw test statistics as conditional probabilities, 21% found the right answer; after learning to restate them as counts out of 1,000, 87% did. The CEO move: require every number that drives a decision to arrive with its denominator, its comparison and its source, and decide in advance which questions are mandatory before acting. The student move: convert every percentage into “how many out of 1,000”, and re-derive one claim a week from its original source.
This guide belongs to the Foundational Cognitive Skills cluster anchored by our piece on deep reading as a competitive advantage. It is the numerical companion to the 5-step protocol for fact-checking AI output and the 12 red flags of bad AI answers. Those posts catch claims that are false. This one handles a harder case: numbers that are true and still mislead.
Why does a non-analyst need statistics now?
Producing a chart or a “most customers prefer” sentence now costs almost nothing, so more numbers reach decision-makers with less checking (see our piece on the signal-to-noise crisis). The latest international measurement of adult numeracy shows how exposed readers are.
The OECD Survey of Adult Skills tested adults aged 16 to 65 in 31 participating countries. The OECD average numeracy score was 263 on a 0 to 500 scale. Adults at Level 1, in the survey’s description, handle whole numbers, decimals, common fractions and percentages and “devise simple strategies using one or two steps”. Reading a relative risk, a conditional probability or a skewed average usually takes more steps than that.
Table 1. Adult numeracy in selected countries, OECD Survey of Adult Skills 2023 (adults aged 16-65)
| Country or economy | Mean numeracy score | At or below Level 1 | Level 4 | Level 5 |
|---|---|---|---|---|
| OECD average | 263 | 25% | 12% | 2% |
| Finland | 294 | 12% | 26% | 5% |
| Japan | 291 | 10% | 21% | 3% |
| Netherlands | 284 | 15% | 20% | 4% |
| Germany | 273 | 20% | 16% | 2% |
| England (UK) | 268 | 21% | 14% | 2% |
| France | 257 | 28% | 10% | 1% |
| Spain | 250 | 30% | 6% | 0% |
| United States | 249 | 34% | 10% | 2% |
| Italy | 244 | 35% | 6% | 0% |
| Chile | 214 | 56% | 2% | 0% |
Source: OECD Survey of Adult Skills 2023, as reprinted by Ireland’s Central Statistics Office (PIAAC 2023 International Comparison, Tables 3.2 and 3.5). Rounded percentages as printed.
Two readings matter. The spread is large: 80 points separate the Finnish and Chilean means, and the low-numeracy share runs from 10% in Japan to 56% in Chile. And even in the strongest country most adults are not at the top: Levels 4 and 5 together cover 31% of Finnish adults and 14% on the OECD average (CEOtudent calculation from rounded shares). Do not assume the person presenting a chart is one of them.
The US National Center for Education Statistics adds a trend: the share of US adults at Level 1 or below in numeracy rose from 29% in 2017 to 34% in 2023, with the caution that 2023 was administered entirely on tablets and the assessments differ.
Concept 1: Is that a relative change or an absolute change?
On a small base, relative changes look dramatic while absolute changes stay tiny.
Gerd Gigerenzer and colleagues, in a 2007 monograph in Psychological Science in the Public Interest, describe the UK contraceptive pill scare. In October 1995 regulators warned that third-generation pills doubled the risk of thrombosis, a 100% increase. The underlying studies showed about 1 case per 7,000 women on the earlier pills and about 2 per 7,000 on the newer ones: an absolute increase of 1 in 7,000. The authors report an estimated 13,000 additional abortions in England and Wales the following year.
The same monograph shows the reverse for benefits. A 25% reduction in breast cancer deaths from mammography screening corresponds, in the authors’ words, to “1 less woman out of 1,000”. In random samples of women in four countries, only 6% gave an answer the authors accepted as correct (0 to 5 in 1,000), while 60% of US respondents believed the benefit was 80 or more in 1,000. (Framing examples from 2007 research, not health guidance.)
Table 2. CEOtudent analysis: the same published result expressed four ways
| Published result | Relative framing | Absolute framing | Counts per 1,000 | Units per one extra or avoided case |
|---|---|---|---|---|
| Thrombosis, newer vs older pill (Gigerenzer et al. 2007) | +100% | +1 in 7,000 (0.014 percentage points) | 0.14 to 0.29 | 7,000 women per extra case |
| Breast cancer deaths with screening (Gigerenzer et al. 2007) | -25% | 1 fewer in 1,000 | about 4 to 3 (implied baseline) | 1,000 women screened per death avoided |
| Tax-rate bar chart (Correll et al. 2020, Fig. 1b) | +13.1% | +4.6 percentage points (35.0% to 39.6%) | 350 to 396 per 1,000 dollars | Bar drawn 5.6 times taller |
CEOtudent analysis of figures in Gigerenzer et al. (2007) and Correll, Bertini and Franconeri (2020); arithmetic in calc.py. The row 2 baseline is derived: a 25% relative cut equal to 1 in 1,000 implies a starting risk of about 4 in 1,000.
The question to ask: “Out of how many?”
Concept 2: What happens out of 1,000? Base rates and natural frequencies
Our post on base rates and the outside view explains why typical outcomes should anchor forecasts. Natural frequencies, probabilities restated as counts in an imagined group, make base rates usable in the moment.
At a 2007 continuing-education session described in Gigerenzer’s monograph, 160 gynecologists were told: 1% of women in a screening population have breast cancer; a woman with cancer tests positive 90% of the time; a woman without cancer still tests positive 9% of the time. Asked what a positive result means, most answered “90%” or “81%”, estimates ranged from 1% to 90%, and 21% chose the best answer.
Restated as counts: ten out of every 1,000 women have breast cancer, and 9 of them test positive. Of the 990 without cancer, about 89 also test positive. So of roughly 98 positives, about 9 have cancer, around 1 in 10 or 9.2% (CEOtudent calculation). After training in this translation, 87% of the gynecologists chose the right answer.
The doctors did not change; the representation did. The method fits any filter, from fraud models to AI-text detectors: start with 1,000 cases, apply the base rate, the hit rate and the false-alarm rate, and count.
The question to ask: “Of everyone flagged, how many are real?”
Concept 3: How big was the sample?
Small groups produce extreme results in both directions because a few cases move their average a lot. Howard Wainer, in a 2007 American Scientist essay, called de Moivre’s 1730 result, that variability of an average shrinks with the square root of the sample size, “the most dangerous equation”.
His central example was expensive. From the late 1990s the Bill and Melinda Gates Foundation backed small schools, with education grants of about $1.7 billion by 2001, partly because small schools were over-represented among top scorers. Wainer and Harris Zwerling examined the 1,662 Pennsylvania schools reporting 5th-grade reading scores. Of the top 50, 6 were among the smallest 3% of schools: 12% instead of the expected 3%, a four-fold over-representation. But 9 of the 50 lowest-scoring schools (18%) were also among the 50 smallest. Small schools crowded both extremes, and the overall relationship between size and 5th-grade scores was essentially flat.
The same applies to workplace league tables: if the best-converting branch or top-rated AI pilot team is also the smallest unit, size alone may explain the rank.
For scale, take an illustrative assumption: a survey question whose true answer is 50%. With 8 respondents the standard error is about 17.7 percentage points, so a rough 95% range spans about 35 points either side. With 400 respondents it is 2.5 points. Fifty times the data buys only about seven times the precision (CEOtudent calculation in calc.py).
The question to ask: “How many cases is that based on, and where do the smallest groups rank?”
Concept 4: Who is missing from the data? Selection filters
Respondents chose to answer, case studies were chosen because they succeeded, studies were published partly because results were striking. When a filter favours certain outcomes, visible data overstates them.
The Open Science Collaboration, with 270 contributing authors and support from the Center for Open Science, repeated 100 experimental and correlational studies published in three psychology journals, using high-powered designs. Its 2015 report in Science found that 97% of the original studies had significant results, against 36% of the replications (35 of 97). The mean effect size fell from 0.403 to 0.197, about half (CEOtudent calculation: 0.49), and in 82 of 99 comparable pairs the original effect was larger.
The figures are specific to the sampled journals, but any literature or “customer stories” page shows the survivors of a filter.
The question to ask: “What is not visible here?” Ask who did not respond, what was excluded and how many attempts preceded the one shown.
Concept 5: Could something else explain it? Correlation and confounders
Two things moving together may share a cause (a confounder), run in reverse, or simply trend together.
A teaching case: in 2012 the New England Journal of Medicine published a short paper by Franz Messerli on a correlation between chocolate consumption and the number of Nobel laureates. As Pierre Maurage, Alexandre Heeren and Mauro Pesenti summarise in The Journal of Nutrition (2013), it led to the suggestion that more chocolate could mean more laureates through cocoa flavanols. They argue the interpretation fails once other flavanol-rich foods are considered, and report high correlations between laureate counts and other measures “whether cogently related or not” (abstract). National wealth and research spending are plausible common causes.
At work the pattern is less amusing. Teams using a new AI tool report higher output, but early adopters may also have had the strongest managers. Customers who use a feature churn less, but engaged customers both use more features and churn less.
The question to ask: “What else differs between these groups, and was anyone randomly assigned?”
Concept 6: Does the pattern reverse when you split the data? Simpson’s paradox
A comparison can point one way in the total and the opposite way inside subgroups when groups are spread unevenly across those subgroups. The classic case is fall 1973 graduate admissions at the University of California, Berkeley, analysed by Peter Bickel, Eugene Hammel and William O’Connell in Science in 1975. Data for the six largest departments ship with the R statistical software, so the numbers can be recomputed exactly.
Table 3. CEOtudent analysis: UC Berkeley admissions, fall 1973, six largest departments
| Department | Men admitted | Men’s rate | Women admitted | Women’s rate | Gap (women minus men) |
|---|---|---|---|---|---|
| A | 512 of 825 | 62.1% | 89 of 108 | 82.4% | +20.3 points |
| B | 353 of 560 | 63.0% | 17 of 25 | 68.0% | +5.0 points |
| C | 120 of 325 | 36.9% | 202 of 593 | 34.1% | -2.9 points |
| D | 138 of 417 | 33.1% | 131 of 375 | 34.9% | +1.8 points |
| E | 53 of 191 | 27.7% | 94 of 393 | 23.9% | -3.8 points |
| F | 22 of 373 | 5.9% | 24 of 341 | 7.0% | +1.1 points |
| All six combined | 1,198 of 2,691 | 44.5% | 557 of 1,835 | 30.4% | -14.2 points |
CEOtudent analysis of the UCBAdmissions dataset distributed with R (4,526 applicants), derived from Bickel, Hammel and O’Connell (1975); rates recomputed in calc.py.
Overall, men were admitted at 44.5% and women at 30.4%. Yet women’s rates were higher in four of the six departments. The reversal comes from where people applied: 51.5% of men applied to departments A and B, which admitted most applicants, while 92.8% of women applied to departments C to F, which rejected most (CEOtudent calculation). The R documentation puts it plainly: the apparent association “stems from differences in the tendency of males and females to apply to the individual departments”.
The question to ask: “Does the result hold inside each segment, and are the groups mixed the same way across segments?”
Concept 7: Was it chosen because it was extreme? Regression to the mean
Anything selected for being unusually high or low tends to measure closer to average next time, intervention or not. Adrian Barnett, Jolieke van der Pols and Annette Dobson (International Journal of Epidemiology, 2005) describe regression to the mean as a phenomenon that “can make natural variation in repeated data look like real change”, strongest when measurement error is large and when follow-up covers only cases selected on an extreme first value (abstract).
Daniel Kahneman’s Nobel autobiography gives the workplace version. A flight instructor objected to his advice on praise: cadets praised for a clean maneuver usually did worse next time, and cadets shouted at usually improved. Kahneman saw that extreme performances are partly luck, which does not repeat on command, so “we are statistically punished for rewarding others and rewarded for punishing them”.
So the worst region improves after a turnaround programme and the best performer slips after a bonus, proving nothing about either.
The question to ask: “Was this picked for an extreme result, and what happened to a similar group with no intervention?”
Concept 8: Average of what? Mean, median and skew
The mean adds everything and divides; the median is the middle case. When a few values are very large, as with income, deal sizes, overruns or time saved by an AI tool, the mean is pulled toward the tail.
The US Census Bureau’s Income in the United States: 2025 (September 2026) puts median household income at $87,460, up 2.6% from $85,210 in 2024. The Bureau’s historical Table H-6 puts the 2025 mean at $126,700: $39,240 or 45% above the median (CEOtudent calculation). The report also notes that income at the 90th percentile was 13.06 times that at the 10th. “The average household” can be off by tens of thousands of dollars without a false number, and so can “average time saved” when a few power users dominate.
The question to ask: “Mean or median, and what does the spread look like?”
Concept 9: Where does the axis start? Misleading chart design
Michael Correll, Enrico Bertini and Steven Franconeri (CHI 2020) reproduce a television bar chart comparing a top tax rate of 35.0% with 39.6%. With the y-axis starting at 34%, the second bar was drawn about six times taller, though the ratio of the values is 1.13 to 1. The paper calls the difference “a 4.6% increase”; strictly it is 4.6 percentage points, a relative rise of 13.1% (CEOtudent calculation). Even careful researchers blur percent and percentage points.
Their crowdsourced experiments found that truncating the y-axis had “a consistent and significant impact on the perceived importance of effect sizes”, in line charts as well as bar charts, and that warning cues such as a broken axis or a gradient fill did not have a consistent effect. Knowing the trick does not neutralise it; you have to read the axis labels.
The question to ask: “What are the axis values, and what change would actually matter here?”
Concept 10: What does “statistically significant” tell you?
On 7 March 2016 the American Statistical Association released six principles on p-values. Principle 2: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” Principle 5: a p-value “does not measure the size of an effect or the importance of a result.” Principle 3 adds that “business or policy decisions should not be based only on whether a p-value passes a specific threshold.”
“Significant” does not mean big, important or likely to replicate, as Concept 4 shows.
A related trap is precision theatre: a productivity gain reported to two decimal places from a pilot with a handful of people. Decimals say nothing about uncertainty. Our audit of the 35,000 decisions a day claim shows a precise-sounding figure circulating for years without a traceable measurement, and our guide to confidence calibration shows how to judge whether a stated range is honest.
The question to ask: “How big is the effect, how uncertain is it, and has anyone else found it?”
The Misleading Number Field Guide
Table 4. CEOtudent editorial framework: the Misleading Number Field Guide
| Concept | The trap in one line | The one question | The honest version | Where it shows up at work |
|---|---|---|---|---|
| Relative vs absolute | Big percentage, tiny base | Out of how many? | “From 4 to 2 per 1,000 (50%)” | Vendor decks, headlines |
| Base rates | Test accuracy read as flag accuracy | Of everyone flagged, how many are real? | True and false positives per 1,000 | Fraud alerts, CV screening, AI detectors |
| Sample size | Small units win and lose rankings | How many cases? | n stated, with a range | League tables, pilots, team surveys |
| Selection | Only survivors visible | What is not visible here? | Response rate, exclusions disclosed | Case studies, testimonials |
| Correlation | Co-movement read as cause | What else differs? Random assignment? | Controlled comparison, confounders named | Adoption vs performance dashboards |
| Simpson’s paradox | Total contradicts the segments | Does it hold in each segment? | Segment table beside the total | Sales by region, ratings by team |
| Regression to the mean | Extremes drift back, action gets credit | Picked for being extreme? Compared with what? | Untreated comparison group | Turnarounds, bonuses, “worst month” pilots |
| Mean vs median | A tail drags the average | Mean or median? Spread? | Median plus percentiles | Pay, time saved, deal size |
| Chart design | Axis manufactures or hides drama | What are the axis values? | Labelled axis, justified scale | Board slides, AI-generated charts |
| Significance and precision | “Significant” and decimals read as importance | How big, how uncertain, replicated? | Effect size with interval | A/B tests, AI summaries of studies |
CEOtudent editorial framework; a reading aid, not a statistical procedure.
A 10-question pre-decision checklist for any statistic
Use it before a number changes a budget, a hire or a strategy.
- Source: Can the original table or paper be opened?
- Denominator: Out of how many?
- Comparison: Compared with what: last year, a control group, nothing?
- Absolute size: If relative, what are the before and after values?
- Base rate: How common is the thing being detected or predicted?
- Sample: How many cases, chosen how?
- Missing data: Who or what is not in the data, and why?
- Alternatives: What else could produce this, including regression to the mean?
- Segments: Does it hold within the main subgroups?
- Uncertainty: What is the range, and has anyone replicated it?
Weekly practice routine (about 30 minutes): Monday, note one statistic from work or an AI answer with its claimed source. Tuesday, find the original, using our fact-checking protocol. Wednesday, convert it into counts out of 1,000 and write the absolute version in one sentence. Thursday, run the ten questions and note which the original presentation skipped. Friday, write two lines on what decision the number should and should not support; as our piece on writing as thinking argues, writing it out tests whether you understood it.
Table 5. Illustrative worked example: a vendor says its AI tool “cuts processing errors by 50%”
| Item | Value | Status |
|---|---|---|
| Transactions per month | 10,000 | Illustrative assumption |
| Current error rate | 4 per 1,000 | Illustrative assumption |
| Error rate after a 50% cut | 2 per 1,000 | Derived from the claim |
| Absolute reduction | 0.2 percentage points; 20 errors per month | Derived |
| Cost per error | 120 currency units | Illustrative assumption |
| Monthly saving | 2,400 | Derived |
| Monthly tool cost | 3,000 | Illustrative assumption |
| Net monthly result | -600 | Derived |
| Break-even current error rate | 5 per 1,000 | Derived |
Every input is an illustrative assumption, not product data; arithmetic in calc.py. The “50%” headline is identical whether the tool pays off or not.
The CEO move and the student move
The CEO move. Make three fields mandatory for any figure that drives a decision: denominator, comparison, source. Decide in writing which checklist questions must be answered before acting, and send back any slide that cannot answer them, whoever or whatever produced it. Pair this with thinking in bets, so uncertain numbers are used as probabilities rather than certainties.
The student move. Train the way the gynecologists did: change the representation until the answer is obvious. Convert every percentage into counts out of 1,000, and re-derive one claim a week from its source. Over months the ten concepts become reflexes, part of the broader evaluation skill of judging output you did not produce.
What this does not mean
- This is not a statistics course. The ten concepts are a defensive minimum; modelling, causal inference and experimental design need training or a specialist.
- Relative numbers are not dishonest. They compare effects across groups well. The problem is a relative number without its absolute base.
- Failed replication does not mean fraud. The Open Science Collaboration found replication success was better predicted by the strength of the original evidence than by characteristics of the teams.
- The health examples are general information about how statistics are communicated, not medical advice; the figures date from the cited studies.
Frequently asked questions
What is the single most useful habit for a non-analyst?
Converting percentages into counts out of 1,000. It exposes relative-versus-absolute confusion, makes base rates visible and moved gynecologists from 21% to 87% correct in the study above.
How can a misleading chart be spotted quickly?
Read the axis labels before the bars. Check where the y-axis starts, whether two series use different scales, and whether the change shown would matter in practice.
Is a bigger sample always better?
Quadrupling a sample halves the margin of error, but a large biased sample is still biased.
What does p < 0.05 mean in plain English?
That the data would be fairly surprising if there were no effect, under the analyst’s model. According to the American Statistical Association it is not the probability that the hypothesis is true, and not a measure of effect size.
How does this apply to numbers produced by AI tools?
The same way, plus one step: confirm the number exists in a real source before analysing it. Our red flags guide covers fabricated figures.
Sources
- Central Statistics Office (Ireland). Programme for the International Assessment of Adult Competencies (PIAAC) 2023: International Comparison, Tables 3.2 and 3.5. Statistical release, 10 December 2024 (reprints OECD Survey of Adult Skills 2023 results).
- US National Center for Education Statistics. Highlights of the 2023 U.S. PIAAC Results: Highlights of U.S. National Results. December 2024.
- Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M. and Woloshin, S. Helping Doctors and Patients Make Sense of Health Statistics. Psychological Science in the Public Interest, 8(2), 53-96, 2007 (University of Konstanz repository copy).
- Wainer, H. The Most Dangerous Equation. American Scientist, 95(3), May-June 2007 (archived copy of the publisher’s page).
- Open Science Collaboration. Estimating the Reproducibility of Psychological Science. Science, 349(6251), aac4716, 2015 (author version, University of Dundee repository).
- American Statistical Association. American Statistical Association Releases Statement on Statistical Significance and P-Values. Press release, 7 March 2016 (lists the six principles; the full statement by Wasserstein and Lazar was not opened).
- Messerli, F. H. Chocolate Consumption, Cognitive Function, and Nobel Laureates. New England Journal of Medicine, 367(16), 1562-1564, 2012 (PubMed record only; content as summarised in source 8).
- Maurage, P., Heeren, A. and Pesenti, M. Does Chocolate Consumption Really Boost Nobel Award Chances? The Peril of Over-Interpreting Correlations in Health Studies. Journal of Nutrition, 143(6), 931-933, 2013 (abstract only).
- R Core Team. UCBAdmissions: Student Admissions at UC Berkeley. R datasets package data file and documentation, citing Bickel, P. J., Hammel, E. A. and O’Connell, J. W., Sex Bias in Graduate Admissions: Data from Berkeley, Science, 187, 1975 (original paper not opened).
- Kahneman, D. Biographical. Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2002, Nobel Prize outreach.
- Barnett, A. G., van der Pols, J. C. and Dobson, A. J. Regression to the Mean: What It Is and How to Deal With It. International Journal of Epidemiology, 34(1), 215-220, 2005 (abstract only; an erratum was published in 2015).
- US Census Bureau. Income in the United States: 2025. Current Population Reports P60-289, September 2026.
- US Census Bureau. Historical Income Tables, Table H-6: Median and Mean Household Income of All Households by Region, 1975 to 2025.
- Correll, M., Bertini, E. and Franconeri, S. Truncating the Y-Axis: Threat or Menace? CHI 2020 Conference on Human Factors in Computing Systems (arXiv version 2, January 2020).
Tables 2, 3 and 5 are CEOtudent analyses (Table 5 is illustrative, every input an assumption); Table 4 is a CEOtudent editorial framework. Combined Level 4 and 5 shares are calculated from rounded published percentages. Deliberately not used: the Charig et al. (1986) kidney-stone subgroup table (only the abstract could be opened), the campus-wide Berkeley totals often quoted (original paper not opened), Pandey et al. (2015) results on deceptive charts (no open full text), and the chocolate-Nobel correlation coefficient (Messerli article not accessible).
This content was compiled with the support of AI following in-depth research, then written and prepared for publication by the CEOtudent editorial team.
This post is also available in:
