1.13: Repeatable experiments increase our confidence.

In 2005, a study showed that some patients being treated for HIV infection who were also taking an epilepsy drug called valproic acid had significantly reduced numbers of HIV particles in their blood. Newspaper headlines announced this finding—“AIDS Cure Possible, Study Suggests”—and raised many people’s hopes. Two years later, however, a study of people who were already taking valproic acid and anti-HIV drugs concluded that those taking valproic acid had not benefited at all from the drug (FIGURE 1-16). It’s not certain why the later study didn’t produce the same results as the first study, but the second study did address two serious shortcomings of the first study—namely, the first study was much smaller, involving only four patients, and no control group was used. This pair of studies reveals the importance of repeatability in science.

A powerful way to demonstrate that observed differences between a treatment group and a control group truly reflect the effect of the treatment is for the researchers to conduct the experiment over and over again. Even better is to have other research groups repeat the experiment and get the same results. Researchers describe this desired characteristic of experiments by saying that an experiment must be “reproducible” and “repeatable.”

An experiment that can be done over and over again by a variety of researchers to give the same results is an effective defense against biases (which we discuss in the next section) and reflects a well-designed experiment. Experiments whose results cannot be confirmed by repeated experiments or by experiments performed by other researchers are the downfall of many dramatic claims. Even though chemist Linus Pauling had not one, but two Nobel prizes, he was never able to convince the scientific community that megadoses of vitamin C are an effective treatment for cancer. Every time other individuals tried to repeat Pauling’s studies, properly matching the control and treatment groups, they found no difference. Vitamin C just isn’t effective against cancer—a conclusion that the vast majority of the medical community now believes. The scientific method is profoundly egalitarian: more important than a scientist’s credentials are sound and reproducible results.

When a study is repeated (also referred to as “replicating” a study), sometimes a tiny variation in the experimental design can lead to a different outcome; this can help us isolate the variable that is primarily responsible for the outcome of the experiment. Alternatively, when experiments are repeated and the same results are obtained, our confidence in them is increased.