Outcome Bias: Why a Good Result Makes a Decision Look Smart

The two versions of the case above contain the same decision. The physician knew the pain, the five extra years of life expectancy and the 8 percent chance of dying on the table, and nothing in either ending adds a fact that was available beforehand. A rater who judged the decision on its information would give both versions the same score. Most groups do not, and always in the same direction: the version that ended well is rated better.

Part of the mechanism is hindsight. Once you know the patient died, the 8 percent reads larger than it did, and the operation looks like a gamble anyone could have seen was bad. People also infer unstated facts from the result: that the surgeon was careless, that the man was frailer than described. The ending acts as evidence about everything else in the case, including the odds that were stated explicitly.

The bias is not a simple logical error, and Jonathan Baron and John Hershey said so themselves in follow-up papers in 1992 and 1995. A decision maker often knows things the evaluator does not, so a run of bad outcomes is genuine evidence about the judgment behind them. Judging by outcomes is warranted when outcomes carry that kind of information and when there are enough of them to average. What the experiments isolate is the case where neither condition holds: one decision, one roll of the dice, and an evaluator handed the whole information set.

The pattern shows up outside the laboratory in places where luck can be measured. Romain Gauriot and Lionel Page examined professional soccer and found that players were rewarded for goals that depended on luck, such as shots that went in off the post, with evaluations that did not separate the quality of the attempt from whether it scored. The practical cost is not that an unlucky decision maker feels unfairly judged. It is that the next decision gets made to avoid blame rather than to get the odds right.

Jonathan Baron and John Hershey published "Outcome bias in decision evaluation" in the Journal of Personality and Social Psychology in 1988. The paper ran a series of experiments on medical and monetary decisions. In the medical set, subjects read short cases in which someone chose between a risky operation and living with a condition, and rated the decision on a scale with a midpoint for a decision that was neither clearly right nor clearly wrong. Each case appeared both with a good outcome and with a bad one, the same subjects rated both, and judged the decision as it stood before the outcome was known.

The bypass case in the demo above is their scenario, including the 8 percent operative mortality, which the authors say was chosen from pilot data to make the decision a hard one. The decision was rated better when the outcome was good. The 2023 replication below recomputed that experiment, which had 20 participants: 0.21 standard deviations where the decision was the patient's own, 0.53 where it was a physician's, 0.90 across all scenarios together. The monetary experiments found the same pattern with gambles: a decision to take a bet was rated better when the bet came in.

A pre-registered replication by Sriraj Aiyer, Hoi Ching Kam, Ka Yuk Ng, Nathaniel Young, Jiaxin Shi and Gilad Feldman, in the International Review of Social Psychology in 2023, ran it again with 692 online participants, each seeing one version only, as the demo above does. The gap came out at 0.77 standard deviations for the patient's own decision and 1.10 for the physician's, larger than the original on a direct test, though the authors caution that the original figures are paired-sample effects and theirs independent-sample ones. It also held among the 44 participants who said outcomes should not be taken into account when a decision is evaluated, where the gap was still about two thirds of a standard deviation. Success also made raters readier to say the outcome should count in the evaluation, and that most people would have gone ahead.

Both papers make the same point about what is measured. The experiments do not show that people cannot tell a good decision from a bad one. They show that an evaluation made after the result is a judgment of the result and the decision mixed together, and that believing they should be separated is not enough to separate them.

Outcome bias arrives with the result, so it is wherever results are reviewed. The rows below are illustrations, not findings from the studies cited on this page.

The outcome cannot be unseen, so the counters work by fixing the evidence in place before it arrives.

• Write the decision down before you act: what you knew, the options, the odds you assigned and what would count as a good result. A review that has this document in front of it is reviewing the decision.
• Mask outcomes when you can. Case reviews, grading and audits can be run with the result hidden until the reasoning has been scored.
• Judge in batches. One decision and one result tell you almost nothing; a hundred decisions made the same way tell you a great deal. This is also the condition under which Hershey and Baron argued that judging by outcomes is warranted.
• Ask the other-ending question out loud. If the patient had lived, or the launch had worked, would this reasoning be in front of a review at all?
• Separate accountability from evaluation. Someone may owe an account of a bad result without the decision having been a bad one, and saying both keeps the review honest.
• Do not expect the right opinion to be enough. In the 2023 replication the gap held among the participants who said outcomes should not count, so the fix has to be in the process rather than in the resolve of the reviewer.

Three neighbors are routinely called outcome bias, and keeping them apart changes what you do about them.

• It is not hindsight bias, the sense that an outcome was predictable all along. Hindsight bias distorts what you think you knew; outcome bias distorts the grade you give a decision, and the first is one of the routes to the second. In the 1988 experiments the outcome moved the ratings even when the stated probabilities were in front of the rater.
• It is not moral luck, which concerns blame and desert rather than decision quality. The drunk driver who hurt nobody is the moral luck case; the physician whose patient died is the outcome bias case.
• It is not survivorship bias, which is about which cases you get to see at all. Outcome bias misjudges the case in front of you; survivorship bias means the failures never reached you.