Survivorship Bias: Why the Bullet Holes You Can See Are the Wrong Ones

Every set of examples you can look at has been filtered by something. Companies you have heard of are mostly the ones that did not fail. Funds in a performance table are the ones still open. Buildings from 1850 that are still standing are the sturdy ones. People who give advice about how they succeeded are, by construction, people who succeeded. If the filter is related to the thing you are trying to learn about, the surviving cases will mislead you, and they will do it with evidence that looks perfectly real.

The bombers are the clearest example because the filter was literal: planes that were hit in the wrong place did not come back to be examined. But the mechanism is the same whenever you reason from the cases that are present to the population that produced them. The absent cases are not merely missing at random. They are missing because of the very outcome you are studying.

Survivorship bias is a form of selection bias with a specific shape: the selection happens on the outcome, and the people doing the reasoning only have access to one side of it. It is unusually hard to notice because the missing data do not announce themselves. Nothing in a list of successful dropouts tells you how many dropouts are not on the list. Correcting it requires asking a question the data cannot answer on their own: who is not here, and why?

The bias also persists because survivors are the easiest cases to find. A book about the habits of billionaires has subjects to interview; the people with the same habits who went bankrupt are harder to locate and less often asked. A fund family that merges its weakest funds into others reports a better average. A university that highlights its famous alumni is not lying. In each case the survivors are real; the inference drawn from them is not.

During the Second World War the Statistical Research Group at Columbia University, which included Abraham Wald, Milton Friedman, Jacob Wolfowitz and Allen Wallis, took on applied problems for the US military. One of them was how to estimate the vulnerability of aircraft to hits on different parts, which bears on where to add armor. The data available were counts of hits by location on aircraft that had returned from missions, and the number of aircraft that had not returned.

Wald's answer, worked out in a series of memoranda in 1943, started from the point that the returning planes were a biased sample: a hit that brought a plane down could not appear in the data. He then built a method to estimate, for each region of the aircraft, the probability that a hit there would be fatal, using the distribution of hits on survivors, the number of planes lost, and an assumption about where hits land; in his worked example each part's expected share of hits is its share of the plane's exposed area. Under that assumption, a region that shows fewer hits on survivors than its exposed area predicts is one where hits tend to be fatal, and that is where protection pays. The memoranda work through hypothetical numerical examples rather than a single wartime map.

The memoranda were classified and circulated inside the group. They were reprinted in 1980 by the Center for Naval Analyses as "A Method of Estimating Plane Vulnerability Based on Damage of Survivors," and in 1984 Marc Mangel and Francisco Samaniego published a restatement in the Journal of the American Statistical Association, "Abraham Wald's work on aircraft survivability," which placed the method within modern estimation theory and gave worked examples.

The story has grown in the retelling. Whether Wald was shown a diagram of red dots, and whether an officer had already proposed armoring the holes, is not documented. What is documented is the method, and the principle that an absent observation can carry more information than a present one.

The pattern appears wherever advice, data or examples come only from the cases that lasted.

The counter is a habit of asking about the cases that are not in front of you.

• Name the filter. Before drawing a conclusion from a set of examples, ask what process produced the set. Who or what had to survive, be published, be remembered or be retained to be included?
• Ask for the denominator. A list of ten successful dropouts is meaningless without the number of dropouts. A win rate needs the number of attempts.
• Seek out the failures directly. Post-mortems, closed-fund databases, rejected applications and retracted claims are where the missing half of the evidence lives.
• Prefer designs that follow everyone from the start. A clinical trial that tracks every enrolled patient, including the ones who quit, is much harder to filter by the outcome than a case series of the patients who finished.
• Be suspicious of advice from the top. The most visible people in any field are the survivors of that field, and the things they did may have been irrelevant or harmful for the people who are not visible.

Survivorship bias is one of several ways a sample can misrepresent a population. Its neighbors differ in where the filtering happens and who is doing it.

• It is not the availability heuristic, which is about judging frequency by how easily examples come to mind. Survivorship bias can feed availability, since survivors are the cases that come to mind, but it is a property of the data rather than of the memory.
• It is not hindsight bias, the sense after the fact that an outcome was predictable. A story that explains a survivor's success in hindsight combines both, but hindsight bias also distorts memory of single events with no selection involved.
• It is not outcome bias, which judges the quality of a decision by how it turned out. Survivorship bias is about which decisions you get to see at all.