Information Bias: Wanting a Test That Cannot Change the Decision

Information feels valuable in itself. Reducing uncertainty is satisfying, and a decision made with a test result behind it feels more defensible than the same decision made without one. That goes wrong when the question of what the information is for drops out. In decision theory, the value of a piece of information is the improvement it makes to the choice you will take, and a test whose every result leads to the same action is worth exactly nothing, whatever it would tell you about the odds.

Baron, Beattie and Hershey described the bias as a missing check. People who are deciding whether to ask a question seem to consider whether the answer would be informative, whether it would support the leading hypothesis, and whether it could settle anything for certain, and these are reasonable things to consider. What they often fail to ask is whether any answer would change the hypothesis they are going to act on. The authors called the habit of valuing questions without that check the information heuristic, and the pattern of ratings it produces information bias. In their interviews, participants who were walked through what they would do after each result frequently saw the point at once; the check is easy to run and simply had not been run.

Two companions appeared in the same experiments: congruence bias, the preference for tests likely to confirm the favored hypothesis over tests that would better distinguish the hypotheses, and certainty bias, the extra value placed on a test that could settle something with 100 percent confidence even when certainty changes nothing. All three come from attending to the question rather than to the decision it serves.

A later line of work found a second cost. Bastardi and Shafir showed in 1998 that when people choose to wait for a piece of information that would not have changed their choice had it been known from the start, they then tend to use it: having pursued the information, they give it weight it does not deserve.

Jonathan Baron, Jane Beattie and John Hershey published "Heuristics and biases in diagnostic reasoning: II. Congruence, information, and certainty" in Organizational Behavior and Human Decision Processes in 1988. The paper reported six experiments on how people evaluate yes-or-no questions when testing hypotheses. In the first three, participants supplied their own probabilities for everyday scenarios; in the last three, the probabilities were given, in abstract medical problems with invented diseases.

The fourth experiment is the one that documents information bias. Participants were asked to evaluate tests for deciding which of three diseases, A, B or C, to treat, with prior probabilities of .64, .24 and .12 that stayed the same in every case. The diseases differed in how likely they were to produce a chemical called tutone, detectable in the blood, and each case gave the probability of tutone for each disease. Participants were told in so many words that whether or not they did the test, they would always treat the most likely disease. They rated the value of each test from 0, worthless, to 100, removes all doubt. Twelve participants' ratings were analyzed.

Only two of the ten cases had any value by the normative model, and those two received the highest ratings, 61 and 75, so participants were sensitive to real value. But two cases that could only distinguish the diseases that would not be treated in any event, B and C, were rated 34 and 26 on average, against 9 for a case in which the test result was equally likely under every disease and carried no information at all. Tests that could change the probability of disease A without ever changing the decision to treat it were also rated above that baseline.

The authors also put two problems to 33 participants in interviews, including the globoma case the demo adapts, in which an ET scan separates two unlikely diseases that will not be treated whatever it shows. A majority answered correctly. The interviews with those who did not showed the heuristic at work, and several saw the error only after being asked what they would actually do with each result.

The bias appears wherever more evidence is available for a choice that is already determined. The rows below are illustrations, not findings from the studies cited on this page.

The counter is the check that the heuristic leaves out, and it can be run in a sentence.

• Before seeking information, list the possible answers and write down what you would do after each one. If every line says the same thing, the information has no value for this decision, whatever its interest.
• Name the result that would change your mind. If you cannot describe one, you are not deciding, you are reassuring yourself, which is a legitimate aim but a different one with its own costs.
• Price the information. Tests, delays and extra interviews cost money, time and sometimes risk; a result that cannot change the action buys nothing with them.
• Be wary of information you have waited for. Bastardi and Shafir's finding is that pursued information gets used even when it should not; a decision that was settled before the wait should be settled after it.
• Separate decision value from other value. Information may matter for research, prognosis or the wish to know; naming those reasons keeps them from being confused with the decision.

Information bias in this sense has close neighbors in the same 1988 paper and a namesake in another field.

• It is not congruence bias, which Baron, Beattie and Hershey documented in the same studies: the preference for tests that would confirm the favored hypothesis over tests that would better distinguish it from the alternatives. Congruence bias picks the wrong test; information bias wants a test when none is needed.
• It is not confirmation bias, the general tendency to seek and weigh evidence that fits what one already believes. Information bias can be entirely even-handed about what the answer will be; its flaw is wanting the answer when the answer cannot matter.
• It is not ambiguity bias, the preference for options whose probabilities are known over options whose probabilities are not. That bias is about which option to choose; information bias is about whether to gather more evidence before choosing.
• It is not information bias in epidemiology, which is a systematic error in the measurement or classification of a study's data, for example when people with a disease recall exposures differently from people without it. The two share nothing but the name.