Insensitivity to Sample Size: Why Small Samples Swing Wider

People judge a sample by how much it looks like the population, and a day with 60 percent boys looks equally unusual whether 15 babies or 45 were born. Kahneman and Tversky called the underlying shortcut the representativeness heuristic: the likelihood of a result is read off its similarity to what is expected, and similarity does not change with the number of cases. Sampling theory does change with it. A proportion computed from 15 cases wanders far more widely around its true value than a proportion computed from 45, and far more again than one computed from 450, because the chance swings in individual cases average out as the count grows.

The arithmetic for the hospitals makes the gap concrete. With 15 births a day and each birth a boy with probability one half, more than 60 percent boys means ten or more of the fifteen, which happens on about 15 percent of days, roughly 55 days in a year. With 45 births, more than 60 percent boys means 28 or more, which happens on under 7 percent of days, roughly 25 days a year. The small hospital records more than twice as many such days, and the intuition that the two should be about equal has no support in the numbers.

The same shortcut produces the belief in what Tversky and Kahneman, in 1971, called the law of small numbers: the expectation that a small sample will reproduce the features of its population, so that a short run of coin flips should look balanced and a small study should show the same effect as a large one. Tversky and Kahneman reported in 1974 that when people were asked to give the distribution of average heights in random samples of 1,000, 100 and 10 men, they produced the same distribution for all three sizes. The error is not confined to novices: the 1971 paper was addressed to research psychologists and their habits of designing small studies and trusting their results.

What makes the error durable is that small samples do often look like their populations, so the shortcut is right more often than not, and that the exceptions are memorable in the wrong way. A hospital day with 70 percent boys is a story; the arithmetic that says such days are routine at a small hospital is not. The counter is not a feeling but a habit: before being impressed by a proportion, ask how many cases it was computed from.

The hospital problem appeared in Daniel Kahneman and Amos Tversky's 1972 paper "Subjective probability: A judgment of representativeness" in Cognitive Psychology, and it is the version printed in their 1974 Science article "Judgment under uncertainty: Heuristics and biases" that most people know. The 1974 text presents it word for word. A certain town is served by two hospitals. In the larger hospital about 45 babies are born each day, and in the smaller hospital about 15 babies are born each day. As you know, about 50 percent of all babies are boys; however, the exact percentage varies from day to day. Sometimes it may be higher than 50 percent, sometimes lower. For a period of one year, each hospital recorded the days on which more than 60 percent of the babies born were boys. Which hospital do you think recorded more such days?

The respondents were undergraduate students, and the three options drew 21 votes for the larger hospital, 21 for the smaller hospital and 53 for about the same, defined as within 5 percent of each other. Fifty-six percent of the 95 students who answered chose the option that sampling theory rules out. Tversky and Kahneman's reading was that the two events are described by the same statistic, more than 60 percent boys, and are therefore equally representative of the general population, so the respondents judged them equally likely. They added that the fundamental notion that a large sample is less likely to stray from 50 percent is evidently not part of people's repertoire of intuitions.

The 1974 article placed the problem among several demonstrations of the same insensitivity. When people were asked to assess the distribution of average height in samples of various sizes, they produced identical distributions, assigning the same probability to an average above six feet in samples of 1,000, 100 and 10 men. In judgments of posterior probability, the article reported, people were far more sensitive to the proportion of red and white balls drawn from an urn than to the number of balls drawn, when the number is what should carry most of the weight. The hospital problem is the memorable case because it states the sample sizes in the question and the answer still ignores them.

The demo on this page changes the format. Instead of three options it asks for two numbers, so that the comparison is between the visitor's own estimates rather than against a majority answer, and so that a visitor who gives the small hospital more days can also see how far each estimate sits from the binomial result. The benchmark assumes exactly 15 and exactly 45 births every day; the problem says about, and real daily counts vary, which would change the numbers slightly without changing which hospital records more.

The error shows up wherever a proportion is quoted without its denominator.

The counter is a question, asked before the proportion has a chance to impress you.

• Ask for the denominator. A percentage, a rate or an average without the number of cases behind it is not yet information. Fifty percent of two is one.
• Expect small samples to swing. The smallest units in any ranking, the smallest schools, towns, clinics or teams, will appear at both the top and the bottom in any given year, through chance alone.
• Use a rough rule for the margin. For a proportion near one half, the margin of error is close to one divided by the square root of the sample size: about 10 percentage points for 100 cases, about 3 for 1,000. A lead inside the margin is not a lead.
• Compare like with like. When judging a small sample, compare it with other samples of the same size rather than with a population figure.
• Be slow to generalize from a run. A streak of five, ten or even twenty outcomes is compatible with a very wide range of underlying rates; a conclusion drawn from it should be held loosely until more cases arrive.
• Run the arithmetic once. Working the hospital problem through, as the demo does, leaves a lasting sense of how much more a sample of 15 wanders than a sample of 45.

Insensitivity to sample size belongs to the representativeness family, and its neighbors are easy to confuse with it.

• It is not the base-rate fallacy, which ignores how common a category is while weighing the specific evidence. Insensitivity to sample size ignores how many cases the evidence rests on. Both come from judging by resemblance, but one drops the prior and the other drops the count.
• It is not the gambler's fallacy, the expectation that a run of one outcome makes the opposite due. The two are cousins: the gambler's fallacy expects a small sample to balance itself out, which is the law of small numbers applied to sequences.
• It is not the clustering illusion, the reading of patterns into random clumps. Insensitivity to sample size is about how much weight to give a result, not about whether a pattern is there.