Illusion of Validity: When Confidence Outruns Accuracy

Confidence in a prediction comes from the fit between the prediction and the evidence in front of you. If the evidence tells a clean story and the forecast is the ending that story seems to call for, the forecast feels solid. Nothing in that feeling consults the one quantity that matters, which is how often predictions made this way turn out to be right. Kahneman and Tversky gave the mismatch its name: the unwarranted confidence produced by a good fit between a predicted outcome and the information it was based on.

Consistency is the strongest producer of fit, and that is where the trap sits. People are more confident predicting the final average of a student whose first-year record is all B grades than of a student whose record mixes A grades and C grades, though the two averages are the same. A consistent record does carry some information, and this page will not pretend otherwise. The problem is that consistency in a set of inputs usually means the inputs are redundant, measuring the same thing several times over, and redundant measurements raise the apparent coherence of the picture much faster than they raise its predictive power. Four highly correlated signals that agree deserve roughly the confidence of one good signal, not four.

Oskamp's experiment is the cleanest measurement of the resulting gap. As judges read more about a single case, their accuracy stopped improving almost immediately while their confidence kept climbing, and the number of answers they changed fell as they went, which suggests a picture forming early and hardening rather than a judgment being refined.

What keeps the illusion in place is the structure of feedback. Einhorn and Hogarth showed that in most judgment tasks the outcomes a forecaster gets to see are selected by the forecast itself: an interviewer learns how the candidates they hired performed and never learns about those they rejected, so the record they accumulate can look like success whatever their actual skill. Where accuracy has been measured against a statistical benchmark, as Dawes, Faust and Meehl summarized for clinical prediction in 1989, simple actuarial formulas have equaled or beaten expert judgment in a long list of domains, while the experts remained confident.

Daniel Kahneman and Amos Tversky published "On the psychology of prediction" in Psychological Review in 1973. The paper's argument is that intuitive prediction works by representativeness: people select the outcome that best matches the evidence and report it as their prediction, with little regard for the base rate of that outcome or for how reliable the evidence is. Their experiments showed predictions that were as extreme as the evidence itself rather than regressing toward the average, which is what a prediction from imperfect information should do.

The illusion of validity is the companion claim about confidence. Having shown that people predict by matching, the authors noted that the confidence people feel follows the same matching process, so that confidence grows with the coherence of the input and not with its diagnostic value. Their grade-record example, the all-B student against the student with A grades and C grades, is presented as an illustration of that point rather than as an experiment with reported figures, and it is best read that way.

The measured version came earlier and from a different field. Stuart Oskamp published "Overconfidence in case-study judgments" in the Journal of Consulting Psychology in 1965. Thirty-two judges, including eight clinical psychologists employed at a state hospital, read a published case history of a man given the pseudonym Joseph Kidd in four successive stages: brief demographic facts, then his childhood, then his school and college years, then his army service and later life. After each stage they answered the same 25 multiple-choice questions about his habits, attitudes and reactions, and rated their confidence in each answer.

Accuracy was poor and flat. No judge ever reached 50 percent correct, the average at the final stage was under 28 percent against a guessing rate of 20 percent, and the difference between the first stage and the last was not statistically significant. Confidence rose from 33 percent to 53 percent across the same four stages. The average gap between confidence and accuracy grew from 7 points to 25. Thirty of the 32 judges became more confident, and the experienced clinicians were no more accurate than the students.

The rows below are illustrations, not findings from the studies cited on this page.

Confidence cannot be talked down, but it can be measured against something.

• Keep a prediction log. Write the forecast, the date and a probability, then score it. A hit rate is the only antidote to a feeling of certainty.
• Count independent signals, not signals. Ask what each piece of evidence measures and set aside the ones that measure the same thing twice.
• Regress toward the average on purpose. A prediction from imperfect evidence should sit between the evidence and the typical outcome, and the weaker the evidence, the closer to typical it belongs.
• Ask for the base rate before reading the case. How often does this outcome happen at all is the number that most predictions should start from.
• Look for the outcomes you do not normally see: the candidates you rejected, the deals you passed on, the patients who went elsewhere. Selective feedback is what keeps the illusion alive.
• Use a formula where one exists. Structured interviews, checklists and simple statistical rules have matched or beaten expert judgment in many domains, which is a reason to use them even when they feel thin.

The illusion of validity is about the confidence attached to a prediction. Several related findings describe other parts of the same picture.

• It is not the overconfidence effect, which is the general finding that stated confidence exceeds accuracy. The illusion of validity names the cause of a particular case of it: confidence produced by how well the evidence fits.
• It is not the Dunning-Kruger effect, which is about how self-assessment varies with skill. The illusion of validity does not depend on skill at all; it was measured in experienced clinicians.
• It is not illusory correlation, the perception of a relationship between two things that are unrelated. That is an error about the data; this is an error about the confidence a prediction deserves.
• It is not the clustering illusion, the sense that a random run contains a pattern. A pattern that is really there can still support a prediction nobody should be confident about.