Experiments
Our goal in choosing a sample is a picture of the population, disturbed as little as possible by the act of gathering information. All observational studies share the principle “observe but don’t disturb.” When Jane Goodall first began observing chimpanzees in Tanzania, she set up a feeding station where the chimps could eat bananas. She later said that was a mistake because it might have changed the apes’ behavior.
In experiments, on the other hand, we want to change behavior. In doing an experiment, we don’t just observe individuals or ask them questions. We actively impose some treatment in order to observe the response. Experiments can answer questions such as “Does aspirin reduce the chance of a heart attack?” and “Do a majority of college students prefer Pepsi to Coke when they taste both without knowing which they are drinking?”
13
Experiments
An experiment deliberately imposes some treatment on individuals in order to observe their responses. The purpose of an experiment is to study whether the treatment causes a change in the response.
EXAMPLE 8 Helping welfare mothers find jobs
The Urban Institute in Washington DC reports that most adult welfare recipients are single mothers in their 20s and 30s with one or two children. Observational studies of welfare mothers show that many are able to increase their earnings and leave the welfare system. Some take advantage of voluntary job-training programs to improve their skills. Should participation in job-training and job-search programs be required of all able-bodied welfare mothers? Observational studies of the current system cannot tell us what the effects of such a policy would be. Even if the mothers studied are a properly chosen sample of all welfare recipients, those who seek out training and find jobs may differ in many ways from those who do not. They are observed to have more education, for example, but they may also differ in values and motivation, things that cannot be observed.
To see if a required jobs program will help mothers escape welfare, such a program must actually be tried. Choose two similar groups of mothers when they apply for welfare. Require one group to participate in a job-training program, but do not offer the program to the other group. This is an experiment. Comparing the income and work record of the two groups after several years will show whether requiring training has the desired effect.
NOW IT’S YOUR TURN
Question 1.2
1.2 Posting lectures on the class website. To determine what students found most helpful, an educational researcher examined student comments from several classes he taught. Mentioned most often was the fact that copies of lectures were posted on the class website for students to download. The researcher recommended that instructors post lecture notes on their course websites. Was this an observational study or an experiment?
The welfare example illustrates the big advantage of experiments over observational studies: in principle, experiments can give good evidence for cause and effect. If we design the experiment properly, we start with two very similar groups of welfare mothers. The individual women of course differ from each other in age, education, number of children, and other respects. But the two groups resemble each other when we look at the ages, years of education, and number of children for all women in each group. During the experiment, the women’s lives differ, but there is only one systematic difference between the two groups: whether or not they are in the jobs program. All live through the same good or bad economic times, the same changes in public attitudes, and so on. If the training group does much better than the untrained group in holding jobs and earning money, we can say that the training program actually causes this happy outcome.
14
One of the big ideas of statistics is that experiments can give good evidence that a treatment causes a response. A big idea needs a big caution: statistical conclusions hold “on the average” for groups of individuals. They don’t tell us much about one individual. If on the average the women in the training program earned more than those who were left out, that says that the program achieved its goal. It doesn’t say that every woman in such a program will be helped. And a big idea may also raise big questions: if we hope the training will raise earnings, is it ethical to offer it to some women and not to others? Chapters 5 and 6 explain how to design good experiments, and Chapter 7 looks at ethical issues.