Confidence Calibration: How to Know How Sure You Should Actually Be

TL;DR. Calibration is the match between how sure you say you are and how often you turn out to be right. In quarterly surveys from June 2001 to March 2011, US chief financial officers gave 80 percent confidence intervals for the next year’s S&P 500 return that contained the realized return only 36 percent of the time. In a three-year geopolitical tournament with Brier-score feedback, forecasters were 65.4 percent sure and 63.3 percent correct. In one lab study, changing the question format alone lifted the hit rate of 90 percent intervals from 73.79 to 88.35 percent. The CEO move: no forecast gets signed without a number, a range and a line in a register that is scored later. The student move: run a 10-question 90 percent interval drill and a 20-claim Brier drill until your hit rates match your stated confidence.

This piece belongs to the CEOtudent mental models cluster. Thinking in bets covers deciding under uncertainty, and the base-rates guide covers where a first estimate should come from. This article covers the step in between: auditing the confidence number itself.

What does it mean to be well calibrated?

A forecaster is calibrated when, across all the occasions they say “70 percent”, the event happens about 70 percent of the time. For a range, the test is the hit rate: how often the stated interval contains the true value. As McKenzie, Liersch and Yaniv put it, well-calibrated people would see 90 percent of their 90 percent intervals contain the true value.

Three errors hide under the word “overconfidence”:

• Overprecision: ranges that are too narrow. Haran, Moore and Morewedge call it the most robust type of overconfidence.
• Overconfident probabilities: saying 90 percent about claims that come true 70 percent of the time.
• Underconfidence: hedging more than the evidence warrants, which shows up in a study of expert intelligence analysts below.

Calibration is not the whole job: someone who always states the base rate can be perfectly calibrated and useless. That second property is called resolution or discrimination; in Bickel, Floehr and Kim’s study of US weather forecasts, the resolution term was much larger than the reliability penalty for miscalibration.

How far apart are stated confidence and realized hit rates?

Table 1. The Calibration Gap (CEOtudent analysis of seven published studies)

Group and task	Stated confidence	Realized hit rate	Gap (points)
US CFOs, 80% intervals for 1-year S&P 500 return, all quarters	80%	36%	+44.0
Same CFOs, least volatile quarters only	80%	59%	+21.0
Online participants, 90% intervals, Washington DC temperature one month ahead	90%	29.17%	+60.8
Lab participants, 90% intervals, election years of US presidents	90%	54.34%	+35.7
Pittsburgh residents, 90% intervals, local temperature one month ahead	90%	73.79%	+16.2
Same residents, 90% intervals built from probability bins (SPIES)	90%	88.35%	+1.7
Internists, easier diagnostic vignettes	7.2 of 10 (read as 72%)	55.3%	+16.7
Internists, harder diagnostic vignettes	6.4 of 10 (read as 64%)	5.8%	+58.2
Geopolitical tournament forecasters, all conditions	65.4%	63.3%	+2.1
Same tournament, teams with one hour of training	64.3%	63.6%	+0.7
Canadian strategic intelligence analysts, 1,514 forecasts	signed index	signed index	-7.6
National Weather Service, 1-day rain probability, cool season, average	20.1%	21.8%	-1.7
The Weather Channel, days it forecast a 70% chance of rain	70%	84%	-14.0
GPT-4 before post-training, multiple-choice test subset	binned	binned	0.7 (unsigned)
GPT-4 after post-training, same subset	binned	binned	7.4 (unsigned)

Sources: Ben-David, Graham and Harvey; Haran, Moore and Morewedge 2010; Meyer et al. 2013 (abstract only); Moore et al. 2016; Mandel and Barnes 2014; Bickel, Floehr and Kim 2011; GPT-4 Technical Report, Figure 8. Gaps computed by CEOtudent in calc.py; positive means overconfident. Reading the internists’ 0-10 confidence scale as a percentage is a CEOtudent rescaling. The intelligence row is the authors’ signed calibration index (minus 0.076). The GPT-4 rows are expected calibration error, an unsigned average.

Read this as description, not a league table: the studies differ in task, sample, period, difficulty and scoring. Four patterns stand out.

1. Ranges fail far more than probabilities. Every interval row except the SPIES one shows 16 to 61 points of overconfidence, while the tournament, intelligence and weather rows sit within 8 points of zero, apart from one probability level at The Weather Channel. Soll and Klayman found the same contrast in one study: choices between two answers showed much less overconfidence than intervals.
2. Format can do what willpower cannot. The same Pittsburgh residents went from a 16.2-point gap to 1.7 points when asked to spread probability across bins instead of naming two endpoints.
3. Difficulty moves accuracy far more than confidence. Meyer and colleagues’ 118 physicians diagnosed 55.3 percent of easier cases and 5.8 percent of harder ones correctly, yet confidence moved only from 7.2 to 6.4 out of 10. Accuracy fell 49.5 points, about 89.5 percent in relative terms; confidence fell about 11.1 percent.
4. Feedback-rich environments end up near zero. Weather forecasters learn the outcome the next day; tournament forecasters saw Brier leaderboards updated as questions closed.

Why are confidence intervals so much worse than yes-or-no probabilities?

Soll and Klayman’s diagnosis is that subjective intervals are systematically too narrow for the accuracy of the information behind them, sometimes only 40 percent as large as needed. The CFO data shows it in market terms: the average CFO interval was 14.5 percent wide, while the spread between the 10th and 90th percentiles of realized one-year S&P 500 returns was 42.2 percent, about 2.91 times larger. Only 3.4 percent of CFOs gave an interval wider than 42.2 percent.

Format is among the most effective levers. Table 2 compares conventional 90 percent intervals with Subjective Probability Interval Estimates (SPIES): the range of possible values is split into bins, each bin gets a probability, and the 90 percent range is read off afterwards.

Table 2. Hit rates of 90 percent intervals by elicitation method (figures as printed)

Experiment	Participants	Quantity estimated	90% interval hit rate	90% SPIES hit rate	SPIES advantage (points, CEOtudent calculation)
1	103 Pittsburgh residents	Pittsburgh high temperature one month ahead	73.79%	88.35%	14.56
2	116 US online participants	Washington DC high temperature one month ahead	29.17%	73.91%	44.74
3	334 Pittsburgh lab participants	Year each 20th-century US president was first elected	54.34%	76.91%	22.57

Source: Haran, Moore and Morewedge, Judgment and Decision Making, 2010. In Experiment 1, intervals built from 5th and 95th fractiles hit the same 73.79 percent as conventional ones, and judges who first made SPIES judgments later drew broader conventional intervals too. Asking for the two bounds separately helps only a little (Haran and colleagues describe Soll and Klayman’s fractile method as a modest reduction in overprecision); forcing attention onto every possible outcome, tails included, is what widens ranges.

Expertise does not fix this. McKenzie, Liersch and Yaniv found IT professionals and students about equally overconfident: experts centred intervals closer to the truth but drew them narrower, and the effects cancelled. Their summary of the literature: 90 percent intervals typically contain the true value about 50 percent of the time.

How much wider? Under an illustrative assumption of normally distributed errors, a 90 percent interval hitting 50 percent needs to be about 2.44 times wider, and the CFOs’ 80 percent intervals hitting 36 percent about 2.74 times, close to the 2.91 ratio above. Treat it as a first correction, not a law.

What does good calibration look like in practice?

Weather forecasters. Bickel, Floehr and Kim verified almost 13 million rain-probability forecasts at 734 stations from 1 November 2008 to 31 October 2010. National Weather Service 1-day forecasts of 0.1 and 0.9 were well calibrated and 0.2 and 0.3 very close. Policy matters: when The Weather Channel said 70 percent, rain fell 84 percent of the time, and the channel avoids 50 percent because it believes users will read it as a lack of knowledge.

Tournament forecasters. Moore and colleagues analysed 494,552 forecasts on 344 questions from 2,860 forecasters over three years of an IARPA-sponsored tournament, scored with the Brier rule. The authors write that an hour of training halves overconfidence over the following year: forecasters working alone went from 4.3 points of overconfidence without training to 2.0 with it, a 53.5 percent reduction, and trained teams reached 0.7.

Intelligence analysts, in the other direction. Mandel and Barnes scored 1,514 strategic forecasts by 15 analysts in the Middle East and Africa division of Canada’s Intelligence Assessment Secretariat. Discrimination was very good (94 percent of forecasts fell on the correct side of 50 percent), but the analysts were underconfident, especially on harder and more policy-relevant forecasts. Making their numbers more extreme cut the mean absolute deviation from perfect calibration from 12.7 percent to 4.2 percent, a 66.9 percent decrease.

The common factor is a scored feedback loop, a theme the CEOtudent guide on developing good judgment explores in depth.

How does a Brier score work, and what counts as good?

The Brier score is the mean squared difference between a probability forecast and the outcome, coded 1 if the event happened and 0 if not. Lower is better: 0 is perfect, and always answering 50 percent on yes-or-no questions scores 0.25. Mandel and Barnes call it a proper scoring rule, so honest reporting is the best strategy.

The useful part is the decomposition, which Mandel and Barnes apply following Murphy’s 1973 partition: Brier score = variance index minus discrimination index plus calibration index. Variance reflects how uncertain the questions were; discrimination rewards separating events from non-events; calibration penalizes gaps between stated and observed frequencies. For the Canadian analysts, 0.240 minus 0.182 plus 0.016 reproduces their published Brier score of 0.074 exactly.

Table 3. Published Brier scores, for orientation only

Forecast set	Brier score	Source
Canadian strategic intelligence, 1,514 forecasts	0.074	Mandel and Barnes 2014
Same, word-only forecasts (“could”, “might”) added at 0.5	0.097	Mandel and Barnes 2014
National Weather Service, 1-day rain probability, cool season (mean square error)	0.090	Bickel, Floehr and Kim 2011, Table 4
Tournament forecasters, last day of each question	0.10	Moore et al. 2016
Meteorologists, next-day rain (Murphy and Winkler 1977)	0.13	as reported by Moore et al. 2016
Tournament forecasters, last week of each question	0.14	Moore et al. 2016
Always answering 50% on a yes-or-no question	0.25	arithmetic

Mandel and Barnes note that the score is affected not only by forecaster skill but also by the uncertainty of the forecasting environment, so scores are not comparable across question sets. Compare yourself with your own past scores on similar questions and with the score of always stating the base rate.

Table 4. A ten-claim worked example (every input is an illustrative assumption)

Bin	Claims	Stated probability	Came true	Hit rate
High confidence	5	90%	4	80%
Moderate confidence	5	60%	3	60%
Result	Brier 0.205	Always 70% (base rate): 0.21	Always 50%: 0.25	Calibration 0.005, discrimination 0.01, variance 0.21

Arithmetic in calc.py. The high bin is 10 points overconfident, so the calibration penalty is small; the real weakness is discrimination, since 0.205 barely beats the 0.21 earned by saying 70 percent about everything. Moving the 90 percent claims to 80 percent would lower the score to 0.200. Read two numbers in every drill: the calibration term says whether to adjust confidence, and the gap to the base-rate score says whether the judgments carry information.

Why do “likely” and “unlikely” fail as numbers?

Sherman Kent, widely regarded as the father of modern intelligence analysis according to Mandel and Barnes, recorded when the problem became obvious. National Intelligence Estimate 29-51 called an attack on Yugoslavia in 1951 a “serious possibility”. Kent meant odds of around 65 to 35 in favour; policy planners had read much lower odds; and among his colleagues on the Board of National Estimates the low man had meant about 20 to 80 and the high 80 to 20. His 1964 essay proposed words with numbers attached. Official scales now exist, but they disagree.

Table 5. The same word, four official meanings (CEOtudent analysis of ICD 203, IPCC guidance, NWS practice and Kent 1964)

Word (ICD 203 term)	US intelligence, ICD 203	IPCC AR5 and AR6 (nearest term)	NWS wording (nearest term)	Kent 1964 (nearest term)	ICD vs IPCC midpoint gap (points)	ICD and IPCC overlap (points)
Likely	55-80%	66-100%	55-74%	“Probable” 63-87%	15.5	14
Very likely	80-95%	90-100%	“Categorical” 75% or more	“Almost certain” 87-99%	7.5	5
Roughly even chance	45-55%	“About as likely as not” 33-66%	“Chance” 25-54%	“Chances about even” 40-60%	-0.5	10
Unlikely	20-45%	0-33%	no term	“Probably not” 20-40%	-16.0	13
Very unlikely	5-20%	0-10%	no term	“Almost certainly not” 2-12%	-7.5	5

Sources: ODNI, ICD 203 (2015, as amended); Mastrandrea et al., IPCC AR5 guidance note (2010), restated as consistent with AR5 in the AR6 Summary for Policymakers (2021); NWS Salt Lake City office review of winter 2007-08; Kent, whose ranges are central values plus or minus his stated tolerance (75 percent, give or take about 12). Midpoints, gaps and overlaps computed in calc.py; a negative gap means the IPCC reading is lower.

ICD 203 categories do not overlap; IPCC categories are nested (“very likely” at 90-100 percent is also “likely”) with what the guidance note calls “fuzzy” boundaries. On “likely” the two share 14 points, about 56 percent of the intelligence range and 41 percent of the IPCC range. The traditional NWS breakdown starts at “slight chance” (15 to 24 percent) and has no term below it.

Table 6. Translation test: the word each system would use for the same probability (CEOtudent analysis)

Probability	ICD 203	IPCC	NWS	Kent 1964
18%	Very unlikely	Unlikely	Slight chance	No term (gap between his categories)
38%	Unlikely	About as likely as not	Chance	Probably not
70%	Likely	Likely	Likely	Probable
85%	Very likely	Likely	Categorical	Probable

Only at 70 percent do all four agree. Readers do not repair the gap: Budescu, Broomell and Por found that people’s numerical readings of IPCC terms deviated significantly from the guidelines even when the readers had the guidelines at hand (abstract only). In Mandel and Barnes’s data, about 16 percent of forecasts carried no number at all, only words such as “could” or “might”. The fix is cheap: write the number next to the word, every time. “Likely (70 percent)” survives being forwarded to someone who learned a different scale.

Are AI models any better calibrated than people?

It depends on their training. Kadavath and colleagues found larger language models well calibrated on diverse multiple-choice and true/false questions in the right format, and models fine-tuned with human feedback seemingly miscalibrated but fairly well calibrated after a simple temperature adjustment. The GPT-4 Technical Report shows the cost of post-training: on a subset of the MMLU dataset, expected calibration error rose from 0.007 for the pre-trained model to 0.074 after post-training, about 10.6 times larger.

Steyvers and colleagues add a further gap, between human confidence in model answers and the models’ own confidence, covered in the CEOtudent guides to red flags in AI answers and trust calibration. A stated confidence, from a colleague or a model, is a claim to be scored.

A calibration training protocol (CEOtudent editorial framework)

The quarterly review of confidence bands is already in the CEOtudent decision journal protocol; these four drills feed it.

Drill 1. Ten-question 90 percent interval drill (weekly, 15 minutes). Pick ten quantities from your field that you can check later (last quarter’s churn, a competitor’s headcount). For each, split the plausible range into five to ten bins, give each bin a probability so the total is 100, tails included, then read off the narrowest range holding 90 percent. Count the hits.

Drill 2. Twenty-claim Brier drill (monthly). Write 20 yes-or-no claims that resolve within 30 to 90 days and give each a probability of 55, 65, 75, 85 or 95 percent. On resolution, compute your Brier score, the always-base-rate score and each band’s hit rate.

Drill 3. Equivalent-bet check (before signing any 90 percent claim). Would you rather win a prize if the claim is true, or by drawing from a box with nine winning tickets and one losing ticket? If you prefer the box, your confidence is below 90 percent; lower the number or widen the range. This is a CEOtudent gut check, not a research-validated method.

Drill 4. Words-to-numbers check (every memo). Circle every probability word, write a number beside it, and print the team’s scale at the top of the document.

Table 7. Self-scoring sheet (CEOtudent editorial framework)

Measure	How to score	Consistent with calibration	Warning sign	First correction
90% interval hit rate (Drill 1)	Hits out of 10	9 or 10	7 or fewer	Widen ranges about 2 times; give tail bins more weight
Brier score (Drill 2)	Mean squared error, 20 claims	Below the always-base-rate score	Above 0.25	Fewer extreme probabilities
Band hit rate (Drill 2)	Share true in the 85-95% bands	Close to the band	Well below it two months running	Move those claims down one band
Equivalent-bet failures (Drill 3)	90% claims where the box won	None	Any	Rewrite the claim or the number
Words without numbers (Drill 4)	Count per memo	Zero	Any	Add the number

Why those Drill 1 thresholds? Someone truly calibrated at 90 percent scores 8 or fewer about 26.4 percent of the time, 7 or fewer about 7.0 percent, and 6 or fewer about 1.3 percent (binomial arithmetic in calc.py). One 8 says little; repeated 7s say a lot. At a 60 percent hit rate, the normal-error assumption suggests widening by about 1.95 times.

The CEO move: put a number and a range on every forecast you sign

Ben-David, Graham and Harvey found more than a forecasting error. Executives miscalibrated about the market showed similar miscalibration about their own firms’ prospects, and firms with miscalibrated executives seemed to follow more aggressive policies, investing more and using more debt financing. That association does not prove cause, but it makes overprecision a business risk rather than a survey quirk.

A forecast register makes the habit concrete: for every forecast that drives a decision, record the claim, resolution date, probability or 90 percent range, the words used in the memo, the owner and, later, the outcome and its Brier contribution. Each quarter, compute hit rates and Brier scores by person and topic. Run a pre-mortem on any plan whose key forecast sits in a band where the record shows overconfidence. The CEOtudent forecast scorecard applies the same scoring to a decade of expert and AI predictions. The student side is the same register read as a curriculum: every miss shows where you are overprecise.

What this does not mean

• Confidence is not bad. The goal is matching, not lowering; the Canadian analysts understated what they knew by being too cautious.
• The gap table does not rank professions. A year-ahead market call and a next-day rain forecast are different tasks.
• A low Brier score does not prove skill. Easy questions produce low scores; compare against a baseline on the same questions.
• Lab results may not transfer fully. The SPIES gains come from temperature and history quizzes, not business forecasts.
• This article is general information about research on judgment; it is not medical, investment or financial advice.

Frequently asked questions

What is confidence calibration in simple terms?
The match between how sure you say you are and how often you are right. It is not accuracy: a forecaster can be calibrated and still uninformative, which is why the Brier decomposition separates the two.

Why are 90 percent ranges so often wrong?
They are drawn too narrow: sometimes only 40 percent as large as needed, according to Soll and Klayman, and typically holding the true value about half the time, according to the literature McKenzie and colleagues summarize.

What is a good Brier score?
There is no universal threshold. A useful forecaster beats 0.25 (always answering 50 percent) and the always-base-rate score on the same questions.

Can calibration be trained?
Yes, with feedback. One hour of training roughly halved tournament forecasters’ overconfidence, and probability bins raised interval hit rates in the lab.

Should AI confidence scores be trusted?
Score them like any stated confidence. The GPT-4 report shows post-training raised calibration error from 0.007 to 0.074 on its test subset.

Sources

1. Ben-David, Itzhak, John R. Graham and Campbell R. Harvey. Managerial Miscalibration. NBER Working Paper 16215, July 2010, revised September 2013; published in the Quarterly Journal of Economics 128(4), 2013.
2. Moore, D. A. et al. Confidence Calibration in a Multiyear Geopolitical Forecasting Competition. Management Science, Articles in Advance, 2016.
3. Haran, U., Moore, D. A. and Morewedge, C. K. A simple remedy for overprecision in judgment. Judgment and Decision Making 5(7), 2010.
4. Soll, J. B. and Klayman, J. Overconfidence in interval estimates. Journal of Experimental Psychology: Learning, Memory, and Cognition 30(2), 2004 (abstract only).
5. McKenzie, Craig R. M., Michael J. Liersch and Ilan Yaniv. Overconfidence in interval estimates: What does expertise buy you? Organizational Behavior and Human Decision Processes 107, 2008.
6. Meyer, A. N. D. et al. Physicians’ diagnostic accuracy, confidence, and resource requests: a vignette study. JAMA Internal Medicine 173(21), 2013 (abstract only).
7. Mandel, D. R. and Barnes, A. Accuracy of forecasts in strategic intelligence. Proceedings of the National Academy of Sciences 111(30), 2014.
8. Bickel, J. E., Floehr, E. and Kim, S. D. Comparing NWS PoP Forecasts to Third-Party Providers. Monthly Weather Review 139, 2011.
9. Graham, R. A Review of PoP Reliability and PoP Bias Trends at WFO SLC for the Winter of 2007-08. National Weather Service, Salt Lake City office.
10. Office of the Director of National Intelligence. Intelligence Community Directive 203, Analytic Standards, signed 2 January 2015, with later technical amendments.
11. Mastrandrea, M. D. et al. Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties. IPCC, 2010.
12. IPCC. Summary for Policymakers. Climate Change 2021: The Physical Science Basis, Sixth Assessment Report, 2021.
13. Kent, S. Words of Estimative Probability. Studies in Intelligence, CIA Center for the Study of Intelligence, 1964; approved for release 1993.
14. Budescu, D. V., Broomell, S. and Por, H.-H. Improving communication of uncertainty in the reports of the Intergovernmental Panel on Climate Change. Psychological Science 20(3), 2009 (abstract only).
15. Kadavath, S. et al. Language Models (Mostly) Know What They Know. arXiv preprint 2207.05221, 2022.
16. OpenAI. GPT-4 Technical Report. arXiv preprint 2303.08774, version 6, 2024.
17. Steyvers, M. et al. What large language models know and what people think they know. Nature Machine Intelligence 7, 2025.

Tables 1, 5, 6 and 7 are CEOtudent analyses or editorial frameworks; Table 4 is an illustrative example built on assumptions; Tables 2 and 3 report figures as printed. Moore and colleagues’ abstract gives about 3 percent overconfidence; the article uses the 2.1-point figure from their results, which matches 65.4 and 63.3 percent. Deliberately not used: the CFO paper’s 36.3 and 36.6 percent variants (its abstract’s 36 percent is used), Murphy and Winkler’s weather paper and Brier’s 1950 paper (not opened; cited through Moore and colleagues, Mandel and Barnes, and Bickel and colleagues), and the training Brier tables already in the CEOtudent decision journal post.

---

This content was compiled with the support of AI following in-depth research, then written and prepared for publication by the CEOtudent editorial team.

This post is also available in: