Hard-Easy Effect: Why Confidence Runs High on Hard Questions

Confidence and accuracy are supposed to track each other. If you are 70 percent sure across a set of answers, about 70 percent of them should be right. The hard-easy effect is the finding that they come apart in a predictable way: on hard questions, stated confidence sits above the proportion correct, and on easy questions it sits below. Plotted against difficulty, calibration slides from underconfidence on easy material to overconfidence on hard material, crossing the point of accuracy somewhere in the middle.

One reason is that people do not adjust their confidence enough when the task changes. Confidence tends to hover in a comfortable band, often somewhere around 70 to 80 percent, whatever the questions are. When the questions are easy and most answers are right, that band is too low. When the questions are hard and answers are near chance, the same band is far too high. The feeling of confidence is relatively insensitive to the actual difficulty, so it overshoots on hard items and undershoots on easy ones.

A second reason is specific to hard questions. When two answers both look plausible, people reach for the reasons that support the one they picked and weigh the reasons against it less, which props up confidence just when accuracy is lowest. On an easy question there is little room for this, because the answer is obvious and confidence has nowhere to inflate; if anything, people hedge a sure thing and come out underconfident.

How large and how real the effect is has been argued over, and this is where the page has to be careful. Gigerenzer, Hoffrage and Kleinbolting showed in 1991 that part of what looks like a difficulty effect comes from how the questions are chosen. When experimenters select hard, tricky items, those are exactly the items where a person's usual cues mislead, so overconfidence appears; when questions are instead a representative sample from a real domain, the overconfidence largely vanishes. Juslin, Winman and Olsson argued in 2000 that the hard-easy effect is substantially a statistical artifact: because any single test is measured with error, the hardest sets will tend to show overconfidence and the easiest underconfidence through regression alone, and once that and the selection of items are controlled, the effect shrinks toward nothing. The pattern is robust in the usual quiz setup; what it means about human judgment is still debated.

Sarah Lichtenstein and Baruch Fischhoff published "Do those who know more also know more about how much they know?" in Organizational Behavior and Human Performance in 1977. Across a series of experiments they had people answer questions and then state, as a probability, how likely each answer was to be correct, which lets confidence be compared directly with the proportion right.

Two findings stood out. The first was general overconfidence: people's stated probabilities ran ahead of their accuracy. In one task, judging whether children's drawings came from Europe or Asia, participants were correct 53 percent of the time, close to chance, but their average confidence was 68 percent. In the same year Fischhoff, Slovic and Lichtenstein found that when people set the odds of being right at 100 to 1, they were correct about 73 percent of the time rather than 99. The second finding was the hard-easy pattern: the overconfidence was largest on the hardest material and shrank, and then reversed into underconfidence, as the questions got easier. Making a separate test out of only easy items, or only hard ones, preserved the pattern at the level of the whole test.

Lichtenstein and Fischhoff also addressed the question in their title. Within a task, people who scored higher were not much better at judging their own accuracy than people who scored lower; knowing more about a subject did not reliably bring better calibration. And training mattered: in related work, groups that got feedback on their confidence, such as weather forecasters who are scored on their stated probabilities every day, were among the best calibrated, which suggested the miscalibration was not fixed.

The later critiques sharpened what the effect does and does not show. Gigerenzer and colleagues in 1991 demonstrated that representative sampling of questions could make the overconfidence disappear, and introduced a theory in which confidence is well calibrated to the cues a person has learned, failing only when experimenters pick items that defeat those cues. Juslin, Winman and Olsson in 2000 showed that regression and item selection together account for much of the hard-easy effect, so that it is weaker, as a claim about biased human judgment, than the early demonstrations implied. Don Moore and Paul Healy's 2008 review separated overconfidence into distinct kinds and placed the hard-easy pattern among them, noting that the direction of the error depends on task difficulty in just the way Lichtenstein and Fischhoff described.

The effect shows up wherever people put confidence on answers of varying difficulty. The rows below are illustrations, not findings from the studies cited on this page.

The effect comes from confidence that does not move enough with difficulty, so the counters tie confidence to evidence rather than to feeling.

• Rate difficulty and confidence separately. Decide first how hard a question is, then set confidence to match, lowering it on the hard ones and raising it on the easy ones.
• Keep a calibration record. Write down how sure you were and whether you were right, across many judgments, and check whether your 80 percents really come true 80 percent of the time.
• Seek feedback on probabilities, not just outcomes. The best-calibrated people, such as weather forecasters, are scored on their stated confidence repeatedly; the scoring is what tunes them.
• On hard calls, ask for the base rate. How often does a prediction this difficult turn out right? That number is a better anchor than the feeling of certainty.
• Argue the other side. Before locking in a confident answer to a hard question, list the reasons it could be wrong; deliberately weighing them offsets the pull toward the answer you chose.
• Remember the easy half. On questions you find obvious, you are probably more right than you feel, so do not hedge a sure thing into a coin flip.

The hard-easy effect is specifically about how the direction of a confidence error flips with difficulty. Several neighbors describe confidence errors that do not flip.

• It is not the overconfidence effect in general, the tendency for stated confidence to exceed accuracy. The hard-easy effect is the finding that this gap depends on difficulty, turning into underconfidence when the task is easy.
• It is not the Dunning-Kruger effect, which is about how the gap between skill and self-assessment changes across people of different ability. The hard-easy effect is about how calibration changes across tasks of different difficulty for the same people.
• It is not illusory superiority, the tendency to rate oneself above average compared with other people. That is a comparison with others; the hard-easy effect needs no comparison, only a question and a confidence rating.