Experimenter's Bias: How a Researcher's Hopes Tilt a Study

A study is a long series of judgment calls, and a researcher usually has a hypothesis before the first one. Each call has some room in it: how to word a question, which measure to treat as the outcome, whether a borderline response counts, which participants to keep, which of several defensible analyses to run, how to describe the result. Any single call is small, and each can be defended on its own. The bias is in the pattern: when the calls are made after the data are seen, they tend, without any intent to deceive, to break in the direction the researcher expected. The expectation does not have to touch a participant to reach the result; it can enter through the researcher's own choices.

This is why experimenter's bias is broader than the better-known route through the subject. A researcher who never meets a participant, who only analyzes a spreadsheet someone else collected, can still tilt the result by choosing the analysis that works. Simmons, Nelson and Simonsohn called these researcher degrees of freedom and showed by simulation that a few of them, each reasonable, multiply the chance of a false positive. The problem is structural, not moral: give an honest person several defensible options and a result they are hoping for, and the options that deliver it will come to seem the right ones.

The oldest documented part of the problem is the route through the subject, and Robert Rosenthal built his career on it. In the photo-rating studies the expectation moved the data without passing through any analysis at all, because the experimenters cued their participants. But Rosenthal's wider point was that the effect generalizes: an expectation can enter wherever a human judgment enters, which in a typical study is nearly everywhere. His remedy, blinding, keeps the expectation away from the people making those judgments by keeping them ignorant of the condition.

The modern remedy adds the analysis. Preregistration, writing down the measure, the sample size, the exclusion rules and the analysis before collecting data, removes the after-the-fact room that researcher degrees of freedom exploit. Neither remedy depends on researchers being more careful or more honest, which is the point: the history of the field is that resolving to be objective does not work, because the biasing happens in choices that feel principled at the time.

Robert Rosenthal and Kermit Fode published "Three experiments in experimenter bias" in Psychological Reports in 1963, one of a series under the heading Psychology of the Scientist. In the core task, experimenters showed participants photographs of faces that had been pre-rated as neutral and asked them to rate how successful each person looked, on a scale from minus ten to plus ten. Some experimenters were told their participants would probably average plus five, a success rating; others were told to expect minus five. All of them read the same script to their participants and were instructed to say nothing else.

Across the three studies, 30 experimenters ran about 375 participants, and the expectation moved the data every time. In each study the lowest mean obtained by any experimenter who expected high ratings was higher than the highest mean obtained by any experimenter who expected low ratings, so the two groups did not overlap at all. A separate experiment that blocked the experimenters from view, to cut visual cues, reduced the effect but did not remove it, which told Rosenthal that both visual and spoken cues were carrying the expectation to the participants. In later work, experimenters even passed their expectations to research assistants who had never been told the hypothesis.

Rosenthal's best-known demonstration ran the same idea with animals rather than faces. In the rat study of 1963, students told their rats had been bred to be good at mazes recorded better maze performance than students told their rats were bred to be poor, though the rats came from the same stock. That result belongs to the page on the observer-expectancy effect, because the expectation there reached the subject. Rosenthal gathered the whole program into his 1966 book Experimenter Effects in Behavioral Research, and with Donald Rubin's 1978 synthesis of 345 studies, established that interpersonal expectancy effects are real and widespread, with a size that varies by setting.

The analytic half of the picture was formalized much later. Joseph Simmons, Leif Nelson and Uri Simonsohn's 2011 paper, "False-positive psychology," showed by simulation that four common liberties in analysis, trying two outcome measures, adding observations after a first look, controlling for a covariate such as gender, and dropping a condition, each raise the false-positive rate on their own, and together lift it from the nominal 5 percent to over 60 percent. They proposed disclosure requirements that became part of the case for preregistration. The two literatures describe the same bias entering at different points: through the participant, and through the analyst.

The bias appears wherever someone who wants a particular answer also makes the choices that produce it. The rows below are illustrations, not findings from the studies cited on this page.

The history of this bias is a history of good intentions failing, so the counters are procedures rather than attitudes. Each one removes room that expectation would otherwise fill.

• Decide in advance and write it down. The outcome measure, the sample size, the exclusion rules and the main analysis should be fixed before the data arrive, so that no later choice can be steered by the result.
• Blind the people who judge. Keep whoever collects the data and scores the outcome ignorant of the condition. Rosenthal's work shows that an expectation reaches a result through the person who holds it.
• Separate the roles. The person with the hypothesis need not be the person who runs the sessions or scores the data, and separating them cuts the path from expectation to measurement.
• Report everything you ran. The false-positive problem comes from showing the analysis that worked and hiding the ones that did not; disclosing all of them lets a reader judge.
• Prefer measures that leave no judgment call. A machine-recorded time or an answer key cannot be nudged, while a rating, a borderline call or a flexible exclusion can.
• Do not rely on resolving to be objective. The whole literature is that this does not work, because the biasing happens in choices that feel principled when they are made.

Experimenter's bias is the umbrella; three nearby terms name parts of it or relatives of it.

• It is not the observer-expectancy effect, which is the single route in which a researcher's expectation reaches the participants through cues and changes their behavior. That is one path inside experimenter's bias, the one Rosenthal's photo and rat studies isolate.
• It is not expectation bias, where what a person expects changes what they themselves perceive and record. There the tilt stays in the measurement; experimenter's bias also covers the design, the exclusions and the analysis, where no perception is involved.
• It is not confirmation bias, the general tendency to seek and weight evidence that fits a belief. Confirmation bias is one of the engines behind experimenter's bias, but experimenter's bias is specifically about how a study comes to support its hypothesis.