2.2 Designing an Experiment

We have previously discussed the importance of using experiments to establish cause and effect relationships between variables. That is the “why” of experiments. In this section, we investigate the “how” of experiments—designing and conducting experiments so that the results are meaningful.

2.2.1 What Is a Randomized, Comparative Experiment?

On November 18, 1976 the California division of the American Cancer Society got nearly one million smokers to quit for the day. That was the beginning of the Great American Smokeout, now held every year on the third Thursday in November. While some people are able to quit smoking “cold turkey,” research has shown that individuals are more successful when they use resources such as support from family and friends, stop-smoking groups, counseling, nicotine patches, or prescription medication.

Researchers in Denmark conducted a study to investigate whether nicotine patches were safe and effective in quitting smoking. In the abstract for an article in the New England Journal of Medicine, they described their methods and results as follows:

The explanatory variable for this study was the type of patch received, and the response variable was whether the individual continued to abstain from smoking, as measured at four intervals up to one year.

This study was an experiment, because there were two treatments involved (the nicotine patch and the placebo patch), and comparative, because the results from the nicotine patch group were compared with those of the placebo group. A placebo is an inactive treatment that has no medical effect, but resembles the active treatment in all other aspects. Here the placebo served as the control treatment, a treatment used to establish the effect of not treating the individuals medically. A study that uses a placebo as a treatment is called placebo-controlled.

In this experiment, the placebo was used in order to determine whether the nicotine in the patch, rather than just having a patch, caused the increase in abstinence from smoking. In a phenomenon known as the placebo effect, some people improve just because they are part of a study or are receiving a treatment, regardless of whether the treatment is real or not.

The researchers described their study as double-blind. This means that all those directly involved with the experiment do not know which individuals are receiving the active treatment, and which are receiving the placebo. This includes not only the subjects of the experiment, but also those administering treatments or interpreting the results. When those conducting the experiment know who is receiving what treatment, they may unintentionally treat the subjects differently in some way.

An experiment is single-blind if either the subjects or the researchers, but not both, do not know who is receiving what treatment. Blinding minimizes the chance of bias, but is not possible in all situations.

The final characteristic of this study is that it is randomized. Recall that randomization was also an important aspect of sample surveys. For sample surveys, randomization involves using impersonal chance to select the sample from the sampling frame. For experiments, the randomization occurs not in sample selection, but rather in the assignment of individuals to treatment groups. This randomization is the best method for creating treatment groups that are, as much as possible, similar in all regards except for the treatment. This allows us to conclude that differences in outcome are due to differences in treatment.

In the results of the study, the researchers state, “Rates of sustained abstinence were significantly better with active treatment than with placebo.” What does “significantly” mean here? The researchers are stating that their results are statistically significant; that is, the differences between treatment groups are so large that they are unlikely to have occurred merely by chance. And if it weren’t chance, then it was probably the difference in treatment—the nicotine in the patch—that caused the differences in the results.

When we looked at sample surveys in the previous section, we wanted to see the nonresponse rate. In the case of experiments, we have similar questions. How many of the individuals who began the study actually completed it? Why did individuals drop out? In this study, two subjects who received the nicotine patch and one who received the placebo dropped out because of side effects.

An easy way to see how an experiment is to proceed is to draw a picture of the design. The picture below shows the design of the nicotine patch experiment.

Design of the Nicotine Patch Experiment

This study is an example of the simplest form of a randomized comparative experiment. The participants are divided into only two treatment groups; if there were three treatments, the random assignment would require dividing the subjects into three groups. The middle part of our picture would then have three lines rather than two, but the random assignment that begins the process, and the comparison of results that end it, would remain the same. In any well-designed experiment, we always begin with random assignment of individuals to treatment groups, and end by comparing the values of the response variable.

The video Snapshots: Experimental Design details how researchers designed an experiment to determine whether nutritional supplements improved knee pain in arthritis sufferers.

It has been shown that some medications work differently in men than in women, so the researchers in this study might have grouped all the women together and all the men together before dividing each gender into treatment groups. We say that such a study has a block design. A block is a group of individuals that share one or more characteristics. Using a block design helps to eliminate the variations caused by the differences between the blocks—in this case, the differences due to sex. The picture below shows a block design for the nicotine patch experiment.

Block Design for the Nicotine Patch Experiment

Human subjects can be assigned to blocks based on traits such as age, ethnicity, educational level or a particular health condition. Block designs are also used in experiments not involving human subjects. In agricultural studies, for example, test plots can be selected based on characteristics like location, soil fertility or precipitation. Two or more treatments are then applied within each test plot, so that differences in outcomes can be attributed to the treatments rather than differences between plots.

As seen in these examples, a well-designed experiment has three critical properties:

1. Control of extraneous influences on the response variable by comparing two or more treatments;
2. Randomization of individuals to treatment groups;
3. Replication of treatment to a number of individuals.

Experiments that lack these properties have conclusions that are open to question.

2.2.2 Matched-Pair Experiments

In order to control for variables not studied, researchers, particularly in the social sciences and education, use matched-pair designs in their experiments. Two individuals, matched according to important characteristics, are paired, with one individual from each pair randomly assigned to the first treatment, and the other to the second treatment. Typically, one of the groups receives the control treatment (a placebo or no treatment), while the second receives the experimental treatment, although two experimental treatments may be administered in some circumstances.

A number of years ago, researchers investigated the question “Does Transcendental Meditation affect grades?” by using sex, college, year, grades, and first letter of last name to match 70 students who took Transcendental Meditation training with students who had not had the training. They compared GPAs one and two quarters after the training, and found no significant difference between those who had the training and those who did not. This is a classic example of a matched-pair design, although in this case the study is an observational one, not an experiment.

Perhaps no set of individuals is as appealing to researchers as subjects for an observational study or an experiment as identical twins. Identical twins, having come originally from a single fertilized egg, are genetically the same, and form a natural matched pair. A BBC News report detailed an experiment in which five-year-old identical twin boys were given different diets for a two-week period. One twin continued to eat his normal diet, while the second consumed only food with no additives. The children were given IQ tests before and after the experiment. While the scores beforehand were identical, after the experiment, the child who had no additives outscored his brother. While this experiment lacked replication (and perhaps randomization as well), it demonstrates the use of matched-pair design as applied to twins—one twin served as the control subject and the other as the experimental subject.

The Vietnam Era Twin Registry is composed of approximately 7,000 male-male twin pairs both of whom served in the military during the time of the Vietnam conflict (1964-1975). A number of studies have been conducted using these pairs of individuals, investigating health issues such as substance abuse, cardiovascular disease and cognitive function. An ongoing study at Tufts University compares the brain function in identical twin pairs, where only one twin was exposed to combat during the Vietnam War. The design of the study involved pairs where the combat-exposed twin experienced post-traumatic stress disorder and control pairs where the combat-exposed twin was healthy

The final way in which to conduct a matched-pair study is to use two measurements on the same individual as the pair. A researcher might pair a measurement before a treatment with one after treatment, or measure the same characteristic by two different methods. Recording a student’s scores on the SAT before and after taking a prep course would be an example of the first type of study, while an individual’s scores from two different IQ tests would be an example of the second.

The StatTutor Lesson – Matched Pairs and Other Block Designs provides an extensive explanation of the properties of block and matched pairs experimental designs.

2.2.3 Ethical Considerations in Studies

Whenever a study involves living subjects, there are questions about the ethics of the design. Are animals being treated humanely, or are human subjects receiving appropriate medical treatment? What level of risk is acceptable in the pursuit of scientific or medical advances?

A medical or health-related study on human subjects is generally referred to as a clinical trial. The federal government has strict guidelines about clinical trials; these are designed to protect the participants in the studies. The website clinicaltrials.gov, a service of the National Institutes of Health, gives a wealth of information about clinical trials in general and about trials in progress in the U.S. and abroad.

One important feature of a clinical trial is informed consent, the process by which the details of the study are explained to a potential participant. These details include the purpose of the study, its length, required procedures, potential benefits, and risks. A person agreeing to participate in the clinical trial signs an informed consent document. The subject is also entitled to further information throughout the study and may withdraw at any time.

But difficulties can arise even from such a beneficial policy. How are improvements made in emergency or trauma care when patients are unable to provide informed consent? In 1996, the U.S. Food and Drug Administration adopted CFR §50.24 (revised in 2006) that regulated exceptions from informed consent for emergency research when the subject, a family member, or legal representative cannot provide consent. This regulation requires that the patient’s condition is life-threatening, available treatments are unproven or unsatisfactory, evidence supports the potential benefit to the patient, and the risks are reasonable with regard to the patient’s condition and both standard and experimental treatments. A further requirement is that the public be notified that such a trial will be taking place in their community, and that they are given a specific procedure for opting out of the trial in advance of any emergency situation. A press release from the University of Michigan Health System explained the federal regulation, and described two studies proposed by UMHS researchers.

Difficulties may also arise when investigators compare an experimental treatment to a placebo, rather than to a standard treatment. Researchers at the University of Chicago examined all clinical asthma trials involving children conducted between 1998 and 2001. They found that 45 of the 70 studies placed children in a placebo group in which they did not receive the established treatment (anti-inflammatory medication). The researchers concluded that because the children were exposed to unnecessary risk, such trials were unethical. In cases in which an accepted treatment exists, this standard treatment should be used as a control rather than a placebo.

Sometimes as a clinical trial proceeds, researchers find evidence that the experimental treatment is causing serious adverse effects, and terminate the study early. In 2002 the National Institutes of Health stopped a study involving 16,000 women taking hormone replacement therapy. The clinical trial was scheduled to continue three more years, but scientists discovered increased risk of breast cancer, strokes, and heart attacks from long-term use of the therapy.

Both experiments and observational studies provide us with data that we can use to investigate questions about the world around us. Each type of study has its advantages and disadvantages. We have noted that experiments allow us to draw conclusions about cause-and-effect relationships, while observational studies yield only conclusions concerning the association between two variables. This would suggest that experiments are preferable, but they are not always possible due to ethical or practical considerations. Whatever the study, the results we obtain are always determined to some degree by chance.