2.3 Explanation: Discovering Why People Do What They Do

It would be interesting to know whether happy people are healthier than unhappy people, but it would be even more interesting to know why. Does happiness make people healthier? Does being healthy make people happier? Does being rich make people healthy and happy? Scientists have developed some clever ways of using their measurements to answer questions like these. In the first section (Correlation), we’ll examine techniques that can tell us whether two variables, such as health and happiness, are related; in the second section (Causation), we’ll examine techniques that can tell us whether this relationship is one of cause and effect; in the third section (Drawing Conclusions), we’ll see what kinds of conclusions these techniques allow us to draw; and in the fourth section (Thinking Critically about Evidence), we’ll discuss the difficulty that most of us have drawing such conclusions.

Correlation

Table : Table 2.1 Hypothetical Data Showing Relationship Between Sleep and Memory

Participant

Hours of Sleep

No. of Presidents Named

A

0  

11

B

0  

17

C

2.7

16

D

3.1

21

E

4.4

17

F

5.5

16

G

7.6

31

H

7.9

41

I

9  

40

J

9.1

35

K

9.6

38

L

9  

43

How much sleep did you get last night? Okay, now, how many U.S. presidents can you name? If you asked a dozen college students those two questions, you’d probably get a pattern of responses like the one shown in TABLE 2.1. Looking at these results, you’d probably conclude that students who got a good night’s sleep tend to be better president-namers than students who pulled an all-nighter.

When you asked college students questions about sleep and presidents, you actually did three things:

35

Researchers have found a positive correlation between mental illness and smoking. Can you think of three reasons why this correlation might exist?
Thinkstock

What’s so cool about this is that simply by looking for synchronized patterns of variation, we can use measurement to discover the relationships between variables. For example, you know that people who smoke generally die younger than people who don’t, but this is just a shorthand way of saying that as the value of cigarette consumption increases, the value of longevity decreases. Correlations not only describe the world as it is, they also allow us to predict the world as it will be. For example, given the correlation between smoking and longevity, you can predict with some confidence that a young person who starts smoking today will probably not live as long as a young person who doesn’t. In short, when two variables are correlated, knowledge of the value of one variable allows us to make predictions about the value of the other variable.

A correlation can be positive or negative. When two variables have a “more-is-more” or “less-is-less” relationship, then they are positively correlated. So, for example, when we say that more sleep is associated with more memory or that less sleep is associated with less memory, we are describing a positive correlation. Conversely, when two variables have a “more-is-less” or “less-is-more” relationship, then they are negatively correlated. When we say that more cigarette smoking is associated with less longevity or that less cigarette smoking is associated with more longevity, we are describing a negative correlation. (See Appendix: Statistics for Psychology for more details on how correlation is measured.)

Causation

We observe correlations all the time: between automobiles and pollution, between bacon and heart attacks, between sex and pregnancy. Natural correlations are the correlations observed in the world around us, and although such observations can tell us whether two variables have a relationship, they cannot tell us what kind of relationship these variables have. For example, many studies (Anderson & Bushman, 2001; Anderson et al., 2003; Huesmann et al., 2003) have found a positive correlation between the amount of violence to which a child is exposed through media such as television, movies, and video games (variable X) and the aggressiveness of the child’s behavior (variable Y). The more media violence a child is exposed to, the more aggressive that child is likely to be. These variables clearly have a relationship—they are positively correlated—but why?

natural correlations

A correlation observed in the world around us.

It isn’t always easy to detect causal relationships accurately. For centuries, people sacrificed their enemies without realizing that doing so doesn’t actually cause rain, and they smoked cigarettes without realizing that doing so actually does cause illness.
The Granger Collection
Apic/Getty Images

The Third-Variable Problem

One possibility is that exposure to media violence (X) causes aggressiveness (Y). For example, media violence may teach children that aggression is a reasonable way to vent anger and solve problems. A second possibility is that aggressiveness (Y) causes children to be exposed to media violence (X). For example, children who are naturally aggressive may be especially likely to seek opportunities to play violent video games or watch violent movies. A third possibility is that a third variable (Z) causes children to be aggressive (Y) and to be exposed to media violence (X), neither of which is causally related to the other. For example, lack of adult supervision (Z) may allow children to get away with bullying others and to get away with watching television shows that adults would normally not allow. In other words, the relation between aggressiveness and exposure to media violence may be a case of third-variable correlation, which means that two variables are correlated only because each is causally related to a third variable. FIGURE 2.2 shows three possible causes of any correlation.

third-variable correlation

Two variables are correlated only because each is causally related to a third variable.

Figure 2.2: FIGURE 2.2 Causes of Correlation If X (exposure to media violence) and Y (aggressiveness) are correlated, then there are at least three possible explanations: X causes Y, Y causes X, or Z (some other factor, such as lack of adult supervision) causes both Y and X, neither of which causes the other.

36

What is third-variable correlation?

How can we determine by simple observation which of these three possibilities best explains the relationship between exposure to media violence and aggressiveness? Take a deep breath. The answer is: We can’t. When we observe a natural correlation, the possibility of third-variable correlation can never be dismissed. But don’t take this claim on faith. Let’s try to dismiss the possibility of third-variable correlation and you’ll see why such efforts are always doomed to fail.

The most straightforward way to determine whether a third variable, such as lack of adult supervision (Z), causes both exposure to media violence (X) and aggressive behavior (Y) is to eliminate differences in adult supervision (Z) among a group of children and see if the correlation between exposure (X) and aggressiveness (Y) is eliminated too. For instance, we could measure children who have different amounts of adult supervision, but we could make sure that for every child we measure who is exposed to media violence and is supervised Q% of the time, we also observe a child who is not exposed to media violence and is supervised Q% of the time, thus ensuring that children who are and are not exposed to media violence have the same amount of adult supervision on average. So if those who were exposed are on average more aggressive than those who were not exposed, we can be sure that lack of adult supervision was not the cause of this difference.

In 1949, Dr. Benjamin Sandler noticed a correlation between the incidence of polio and ice cream consumption, and he concluded that sugar made children susceptible to the disease. Public health officials issued warnings. As it turned out, a third variable—warm weather—caused both an increase in disease (viruses become more active in the summer) and an increase in ice cream consumption.
Bettmann/Corbis
George Marks/Getty Images

But don’t applaud just yet. Because even if we used this technique to eliminate a particular third variable (such as lack of adult supervision), we would not be able to dismiss all third variables. For example, as soon as we finished making these observations, it might suddenly occur to us that emotional instability could cause children to gravitate toward violent television or video games and to behave aggressively. Emotional instability would be a new third variable (Z), and we would have to design a new test to investigate whether this variable explains the correlation between exposure (X) and aggression (Y). Unfortunately, we could keep dreaming up new third variables all day long without ever breaking a sweat, and every time we dreamed one up, we would have to rush out and do a new test to determine whether this third variable was the cause of the correlation between exposure and aggressiveness. Because we can’t rule out every possible third variable, we can never be absolutely sure that the correlation we observe between X and Y is evidence of a causal relationship between them. The third-variable problem refers to the fact that a causal relationship between two variables cannot be inferred from the naturally occurring correlation between them because of the ever-present possibility of third-variable correlation. In other words, if we care about causality, then naturally occurring correlations just won’t tell us what we really want to know. Luckily, another technique will. That technique is called an experiment, which is a technique for establishing the causal relationship between variables. The best way to understand how an experiment eliminates all possible third variables is by examining its two key features: manipulation and random assignment.

third-variable problem

The fact that a causal relationship between two variables cannot be inferred from the naturally occurring correlation between them because of the ever-present possibility of third-variable correlation.

experiment

A technique for establishing the causal relationship between variables.

37

Where should you put the “sign up” button on a Web page? Developers often create two versions of their Web pages. Some visitors see one version, some see the other, and the developer measures the number of clicks that each version elicits. Developers call this “A/B Testing,” but scientists just call it experimentation.

Manipulation

The most important thing to know about experiments is that you’ve been doing them all your life. Imagine that you are scanning Facebook on a laptop when all of a sudden you lose your wireless connection. You suspect that another device—say, your roommate’s new smartphone—has somehow bumped you off the network. What would you do to test your suspicion? Observing a natural correlation wouldn’t help much. You could carefully note when you did and didn’t have a connection and when your roommate did and didn’t use his phone, but even if you observed a correlation between these two variables, you still couldn’t conclude that the phone was causing you to lose your network connection. After all, if your roommate was afraid of loud noises and called his mommy for comfort whenever there was an electrical storm, and if that storm interfered with your wireless network, then the storm (Z) would be the cause of both your roommate’s phone usage (X) and your connectivity problem (Y).

So how could you test your suspicion? Well, rather than observing the correlation between phone usage and connectivity, you could try to create a correlation by intentionally making calls on your roommate’s phone and observing changes in your laptop’s connectivity as you did so. If you observed that “wireless connection off” only occurred in conjunction with “phone on,” then you could conclude that your roommate’s phone was the cause of your failed connection, and you could sell the phone on eBay and then lie about it when asked. The technique you used to solve the third-variable problem is called manipulation, which involves changing a variable in order to determine its causal power.

manipulation

Changing a variable in order to determine its causal power.

How does manipulation solve the third-variable problem?

How do you determine whether eating 60 hot dogs will make you sick? You eat them one day, don’t eat them the next day, and then see which day you barf. That’s manipulation! BTW, in 2013, world champion Joey Chestnut ate 69 hot dogs in 10 minutes by folding them up.That’s manipulation too!
AP Photo/Henny Ray Abrams

Manipulation can solve scientific problems too. For example, imagine that our passive observation of children revealed a positive correlation between exposure to violence and aggressive behaviors, and that now we want to design an experiment to determine whether the exposure is the cause of the aggression. We could ask some children to participate in an experiment, have half of them play violent video games for an hour while the other half does not, and then, at the end of the hour, we could measure their aggression and compare the measurements across the two groups. When we compared these measurements, we would essentially be computing the correlation between a variable that we manipulated (exposure) and a variable that we measured (aggression). But because we manipulated rather than measured exposure, we would never have to ask whether a third variable (such as lack of adult supervision) caused children to experience different levels of exposure. After all, we already know what caused that to happen. We did!

38

Experimentation involves three critical steps (and several ridiculously confusing terms):

Figure 2.3: FIGURE 2.3 Manipulation The independent variable is exposure to media violence and the dependent variable is aggression. When we compare the behavior of participants in the experimental and control groups, we are essentially computing the correlation between the independent variable and the dependent variable. Notice how the chart to the right looks just like the chart shown in Table 2.1.

Random Assignment

Why can’t we allow people to select the condition of the experiment in which they will participate?

Once we have manipulated an independent variable and measured a dependent variable, we’ve done one of the two things that experimentation requires. The second thing is a little less intuitive but equally important. Imagine that we began our exposure and aggression experiment by finding a group of children and asking each child whether he or she would like to be in the experimental group or the control group. Imagine that half the children said that they’d like to play violent video games and the other half said they would rather not. Imagine that we let the children do what they wanted to do, measured aggression some time later, and found that the children who had played the violent video games were more aggressive than those who had not. Would this experiment allow us to conclude that playing violent video games causes aggression? Definitely not—but why not?

What are the three main steps in doing an experiment?

39

There is no evidence that Louise Hay’s techniques can cure cancer. But even if cancer victims who bought her books did show a higher rate of remission than those who didn’t, there would still be no evidence because buyers are self-selected and thus may differ from nonbuyers in countless ways.
Ingram Publishing/Getty Images

Because we let the children decide for themselves whether or not they would play violent video games, and children who ask to play such games are probably different in many ways from those who ask not to. They may be older, or stronger, or smarter—or younger, weaker, or dumber—or less often supervised or more often supervised. The list of possible differences goes on and on. The whole point of doing an experiment was to divide children into two groups that differed in only one way— namely, in terms of their exposure to media violence. The moment we let the children decide for themselves whether they would be in the experimental group or the control group, we had two groups that differed in countless ways, and any of those countless differences could have been responsible for any differences we observed in their aggression. Self-selection is a problem that occurs when anything about a person determines whether he or she will be included in the experimental or control group.

self-selection

A problem that occurs when anything about a person determines whether he or she will be included in the experimental or control group.

If we want to be sure that there is one and only one difference between the children in our study who are and are not exposed to media violence, then their inclusion in the experimental or control groups must be randomly determined. One way to do this is to flip a coin. For example, we could walk up to each child in our experiment, flip a coin, and assign the child to play violent video games if the coin lands heads up and not to play violent video games if the coin lands heads down. Random assignment is a procedure that lets chance assign people to the experimental or the control group.

random assignment

A procedure that lets chance assign people to the experimental or control group.

What would happen if we assigned children to groups with a coin flip? As FIGURE 2.4 shows, we could expect the experimental group and the control group to have roughly equal numbers of supervised kids and unsupervised kids, roughly equal numbers of emotionally stable and unstable kids, roughly equal numbers of big kids and small kids, of active kids, fat kids, tall kids, funny kids, and kids with blue hair named Harry McSweeny. Because the kids in the two groups would be the same on average in terms of height, weight, emotional stability, adult supervision, and every other variable in the known universe except the one we manipulated, we could be sure that the variable we manipulated (exposure) was the one and only cause of any changes in the variable we measured (aggression). After all, if exposure was the only difference between the two groups of children then it must be the cause of any differences in aggression we observe.

Figure 2.4: FIGURE 2.4 Random Assignment Children with adult supervision are shown in orange and those without adult supervision are shown in blue. The independent variable is exposure to media violence and the dependent variable is aggression. Random assignment ensures that participants in the experimental and the control groups are on average equal in terms of all possible third variables. In essence, it ensures that there is no correlation between a third variable and the dependent variable. You can see this clearly if you compare the chart to the right with the chart shown in FIGURE 2.3.

40

Why is random assignment so useful and important?

Do strawberries taste better when dipped in chocolate? If you dip the big juicy ones and don’t dip the small dry ones, then you won’t know if the chocolate is what made the difference. But if you randomly assign some to be dipped and others not to be dipped, and if the dipped ones taste better on average, then you will have demonstrated scientifically what every 3-year-old already knows.
Robert Daly/Getty Images

Significance

Random assignment is a powerful tool, but like a lot of tools, it doesn’t work every time, we use it. If we randomly assigned children to play or not to play violent video games, we could expect the two groups to have roughly equal numbers of supervised and unsupervised kids, roughly equal numbers of emotionally stable and unstable kids, and so on. The key word in that sentence is roughly. Coin flips are inherently unpredictable, and every once in a long while by sheer chance alone, a coin will assign more unsupervised, emotionally unstable kids to play violent video games and more supervised, emotionally stable kids to play none. When this happens, random assignment has failed—and when random assignment fails, the third-variable problem rises up out of its grave like a guy with a hockey mask and a grudge.

How can we tell when random assignment has failed? Unfortunately, we can’t for sure. But we can calculate the odds that random assignment has failed each time we use it. Psychologists perform a statistical calculation every time they do an experiment, and they do not accept the results of those experiments unless the calculation tells them that there is less than a 5% chance that they would have found differences between the experimental and control groups if random assignment had failed. Such differences are said to be statistically significant, which means they were unlikely to have been caused by a third variable.

Drawing Conclusions

If we applied all the techniques discussed so far, we would have designed an experiment that had internal validity, which is an attribute of an experiment that allows it to establish causal relationships. When we say that an experiment is internally valid, we mean that everything inside the experiment is working exactly as it must in order for us to draw conclusions about causal relationships. But what exactly are those conclusions? If our imaginary experiment revealed a difference between the aggressiveness of children in the experimental and control groups, then we could conclude that media violence as we defined it caused aggression as we defined it in the people whom we studied. Notice those phrases in italics. Each corresponds to an important restriction on the kinds of conclusions we can draw from an experiment, so let’s consider each in turn.

internal validity

An attribute of an experiment that allows it to establish causal relationships.

Representative Variables

The results of any experiment depend, in part, on how the independent and dependent variables are defined. For instance, we are more likely to find that exposure to media violence causes aggression when we define exposure as “playing Grand Theft Auto for 10 hours” rather than “playing Pro Quarterback for 10 minutes,” or when we define aggression as “interrupting another person” rather than “smacking someone silly with a tire iron.” The way we define variables can have a profound influence on what we find, so which of these is the right way?

41

The Real World: Oddsly Enough

Oddsly Enough

A recent Gallup survey found that 53% of college graduates believe in extra-sensory perception, or ESP. Very few psychologists share that belief. What makes them such a skeptical lot is their understanding of the laws of probability.

The Nobel laureate Luis Alvarez was reading the newspaper one day, and a particular story got him thinking about an old college friend whom he hadn’t seen in years. A few minutes later, he turned the page and was shocked to see the very same friend’s obituary. But before concluding that he had an acute case of ESP, Alvarez decided to use probability theory to determine just how amazing this coincidence really was.

First he estimated the number of friends an average person has, and then he estimated how often an average person thinks about each of those friends. With these estimates in hand, he did a few simple calculations and determined the likelihood that someone would think about a friend five minutes before learning about that friend’s death. The odds were astonishing. In a country the size of the United States, for example, Alvarez predicted that this amazing coincidence should happen to 10 people every day (Alvarez, 1965).

Michael Maslin ©The New Yorker Collection/www.cartoonbank.com

“In 10 years there are 5 million minutes,” says statistics professor Irving Jack. “That means each person has plenty of opportunity to have some remarkable coincidences in his life” (quoted in Neimark, 2004). For example, 250 million Americans dream for about two hours every night (that’s a half billion hours of dreaming!), so it isn’t surprising that two people sometimes have the same dream, or that we sometimes dream about something that actually happens the next day. As mathematics professor John Allen Paulos (quoted in Neimark, 2004) put it, “In reality, the most astonishingly incredible coincidence imaginable would be the complete absence of all coincidence.”

If all of this seems surprising to you, then you are not alone. Research shows that people routinely underestimate the likelihood of coincidences happening by chance (Diaconis & Mosteller, 1989; Falk & McGregor, 1983; Hintzman, Asher, & Stern, 1978). If you want to profit from this fact, assemble a group of 24 or more people, and bet anyone that at least two of the people share a birthday. The odds are in your favor, and the bigger the group, the better the odds. In fact, in a group of 35, the odds are 85%. Happy fleecing!

Does piercing make a person more or less attractive? The answer, of course, depends entirely on how you operationally define piercing.
© Radius Images/Alamy
AP Photo/Keystone, TIPress/Samuel Golay

One answer is that we should define variables in an experiment as they are defined in the real world. External validity is an attribute of an experiment in which variables have been defined in a normal, typical, or realistic way. It seems pretty clear that the kind of aggressive behavior that concerns teachers and parents lies somewhere between an interruption and an assault, and that the kind of media violence to which children are typically exposed lies somewhere between sports and felonies. If the goal of an experiment is to determine whether the kinds of media violence to which children are typically exposed cause the kinds of aggression with which societies are typically concerned, then external validity is essential. When variables are defined in an experiment as they typically are in the real world, we say that the variables are representative of the real world.

external validity

An attribute of an experiment in which variables have been defined in a normal, typical, or realistic way.

42

Hot Science: Do Violent Movies Make Peaceful Streets?

Do Violent Movies Make Peaceful Streets?

Quentin Tarantino is a director known for making violent movies, such as Kill Bill 2. Did seeing this movie cause anyone to commit murder? We can’t say for sure, but one thing we do know is that while people were busy watching the movie, they weren’t busy shooting each other.
Miramax/Photofest

In 2000, the American Academy of Pediatrics and five other public health organizations issued a joint statement warning about the risks of exposure to media violence. They cited evidence from psychological experiments in which children and young adults who were exposed to violent movie clips showed a sharp increase in aggressive behavior immediately afterward. These health organizations noted that “well over 1,000 studies … point overwhelmingly to a causal connection between media violence and aggressive behavior” (American Academy of Pediatrics, 2000).

Given the laboratory results, we might expect to see a correlation in the real world between the number of people who see violent movies in theaters and the number of violent crimes. When economists Gordon Dahl and Stefano Della Vigna (2009) analyzed crime statistics and box office statistics, they found just such a correlation—except that it was negative! In other words, on evenings when more people went to the theater to watch violent movies, there were fewer violent crimes. Why? The researchers suggested that violent movies are especially appealing to the people who are most likely to commit violent crimes. Because those people are busy watching movies for a few hours, violent crime drops. In other words, blood-and-bullet movies take criminals off the street by luring them to the theater!

Laboratory experiments clearly show that exposure to media violence can cause aggression. But as the movie theater data remind us, experiments are a tool for establishing the causal relationships between variables and are not meant to be miniature versions of the real world, where things are ever so much more complex.

External validity sounds like such a good idea that you may be surprised to learn that most psychology experiments are externally invalid—and that most psychologists don’t mind. The reason for this is that psychologists are rarely trying to learn about the real world by creating tiny replicas of it in their laboratories. Rather, they are usually trying to learn about the real world by using experiments to test hypotheses derived from theories, and externally invalid experiments can often do that quite nicely (Mook, 1983). For example, the idea that exposure to media violence causes aggression (a theory) suggests that children in a laboratory should behave more aggressively after being exposed to media violence (a hypothesis), and thus their behavior in the laboratory serves to test that theory (see the Hot Science box, Do Violent Movies Make Peaceful Streets?).

Why isn’t external validity always necessary?

Representative People

Our imaginary experiment on exposure to media violence and aggression would allow us to conclude that exposure as we defined it caused aggression as we defined it in the people whom we studied. That last phrase represents another important restriction on the kinds of conclusions we can draw from experiments.

Who are the people whom psychologists study? Psychologists rarely observe an entire population, which is a complete collection of people, such as the population of human beings (about 7 billion), the population of Californians (about 38 million), or the population of people with Down syndrome (about 1 million). Rather, psychologists observe a sample, which is a partial collection of people drawn from a population.

population

A complete collection of participants who might possibly be measured.

sample

A partial collection of people drawn from a population.

43

What is the difference between a population and a sample?

When Jay Greenburg was just 15 years old, his first CD was released and his Violin Concerto premiered at Carnegie Hall.
AP Photo/Sony, Bill Phelps

The size of a sample can be as small as 1. For example, some individuals are so remarkable that they deserve close study, and psychologists study them by using the case method, which is a procedure for gathering scientific information by studying a single individual. We can learn a lot about memory by studying Akira Haraguchi, who can recite the first 100,000 digits of pi, or about creativity by studying Jay Greenburg whose 5th symphony was recorded by the London Symphony Orchestra when he was just 15 years old. Cases like these are not only interesting in their own right, but can also provide important insights into how the rest of us work.

case method

A procedure for gathering scientific information by studying a single individual.

Of course, most of the psychological studies you will read about in this book included samples of ten, a hundred, a thousand, or even several thousand people. So how do psychologists decide which people to include in a sample? One way to select a sample from a population is by random sampling, which is a technique for choosing participants that ensures that every member of a population has an equal chance of being included in the sample. When we randomly sample participants from a population, the sample is said to be representative of the population. Random sampling allows us to generalize from the sample to the population—that is, to conclude that what we observed in our sample would also have been observed if we had measured the entire population. You probably already have solid intuitions about the importance of random sampling. For example, if you stopped at a farm stand to buy a bag of cherries and the farmer offered to let you taste a few that he had handpicked from the bag, you’d be reluctant to generalize from that sample to the population of cherries in the bag. But if the farmer invited you to pull a few cherries from the bag at random, you’d probably be willing to take those cherries as representative of the cherry population.

random sampling

A technique for choosing participants that ensures that every member of a population has an equal chance of being included in the sample.

Generalizing from nonrandom samples can lead to mistaken conclusions. In the presidential election of 1948, the Chicago Tribune mistakenly predicted that Thomas Dewey would beat Harry Truman. Why? Because polling was done by telephone, and Dewey Republicans were more likely to have telephones than were Truman Democrats. In the presidential election of 2004, exit polls mistakenly predicted that John Kerry would beat George Bush. Why? Because polling was done by soliciting voters as they left the polls, and Kerry supporters were more willing to stop and talk.
ULLSTEIN BILD/The Granger Collection
REUTERS/Jim Young

Random sampling sounds like such a good idea that you might be surprised to learn that most psychological studies involve nonrandom samples—and that most psychologists don’t mind. Indeed, virtually every participant in every psychology experiment you will ever read about was a volunteer, and most were college students who were significantly younger, smarter, healthier, wealthier, and Whiter than the average Earthling. About 96% of the people whom psychologists study come from countries that have just 12% of the world’s population, and 70% come from the United States alone (Henrich, Heine, & Norenzayan, 2010). This is because most psychology experiments are conducted by professors and graduate students at colleges and universities in the Western hemisphere, and as much as they might like to randomly sample the population of the planet, they are pretty much stuck studying the folks who volunteer for their studies.

44

So how can we learn anything from psychology experiments? Isn’t the failure to sample randomly a fatal flaw? No, it’s not, and there are three reasons why.

Why is nonrandom sampling not a fatal flaw?

This mouse died after drinking the green stuff. Want to drink the green stuff? Why not? You’re not a mouse, are you?
David J. Green/Alam
  1. Sometimes the similarity of a sample and a population doesn’t matter. If one pig flew over the Statue of Liberty just one time, it would instantly disprove the traditional theory of porcine locomotion. It wouldn’t matter if all pigs flew or if any other pigs ever flew. One flying pig is enough. An experimental result can be illuminating even when the sample isn’t typical of the population.
  2. When the ability to generalize an experimental result is important, psychologists perform new experiments that use the same procedures with different samples. For example, after measuring how a nonrandomly selected group of American children behaved after playing violent video games, we might try to replicate our experiment with Japanese children, or with American teenagers, or with deaf adults. If the results of our study were replicated in these other samples, then we would be more confident (but never completely confident) that the results describe a basic human tendency.
  3. Sometimes the similarity of the sample and the population is a reasonable starting assumption. Few of us would be willing to take an experimental medicine if a nonrandom sample of seven participants took it and died, and that would be true even if the seven participants were mice. Although these nonrandomly sampled participants would be different from us in many ways (including tails and whiskers), most of us would be willing to generalize from their experience to ours because we know that even mice share enough of our basic biology to make it a good bet that what harms them can harm us too. By this same reasoning, if a psychology experiment demonstrated that some American children behaved violently after playing violent video games, we should ask whether there is any compelling reason to suspect that Ecuadorian college students or middle-aged Australians would behave any differently? If the answer is yes, then experiments provide a way for us to investigate that possibility.

Thinking Critically about Evidence

As you’ve seen in this chapter, the scientific method produces empirical evidence. But empirical evidence is only useful if we know how to think about it, and the fact is that most of us don’t. Using evidence requires critical thinking, which involves asking ourselves tough questions about whether we have interpreted the evidence in an unbiased way, and about whether the evidence tells not just the truth, but the whole truth. Research suggests that most people have trouble doing both of these things and that educational programs designed to teach or improve critical thinking skills are not particularly effective (Willingham, 2007). Why do we have so much trouble thinking critically? There are two reasons: First, we tend to see what we expect to see, and second, we tend to ignore what we can’t see. Let’s explore each of these tendencies in turn.

45

We See What We Expect to See

People’s beliefs can color their views of evidence and cause them to see what they expected to see. For instance, participants in one study (Darley & Gross, 1983) learned about a little girl named Hannah. One group of participants was told that Hannah came from an affluent family, which led the participants to expect her to be a good student. Another group of participants was told that Hannah came from a poor family, which led them to expect her to be a bad student. All participants were then shown a video of Hannah taking a reading test and were asked to rate Hannah’s performance. Although both groups saw exactly the same video, those participants who believed that Hannah was from an affluent family rated her performance more positively than did those who believed that Hannah was from a poor family. What’s more, both groups of participants defended their conclusions by citing evidence from the video. Numerous experiments show that people tend to look for evidence that confirms their beliefs (Hart et al., 2009; Snyder & Swann, 1978) and tend to stop looking when they find it (Kunda, 1990). What’s more, when people do encounter evidence that disconfirms their beliefs, they hold it to a very high standard (Gilovich, 1991; Ross, Lepper, & Hubbard, 1975).

How do our beliefs shape the way we think about evidence?

Because it is so easy to see what we expect to see, the first step in critical thinking is simply to doubt your own conclusions—and one the best ways to doubt yourself is to talk to other people! Scientists go out of their way to expose themselves to criticism by sending their papers to the colleagues who are most likely to disagree with them or by presenting their findings to audiences full of critics, and they do this mainly so they can achieve a more balanced view of their own conclusions. If you want to be happy, take your friend to lunch; but if you want to be right, take your enemy.

Sir Francis Bacon (1561–1626) not only devised the scientific method, but also identified the two tendencies that make critical thinking so difficult. He bemoaned the fact that “human understanding, once it has adopted opinions … draws everything else to support and agree with them,” and also that “little or no attention is paid to things invisible.”
Stock Montage/Getty Images

We Consider What We See and Ignore What We Don’t

Not only do people see what they expect to see, but they fail to consider what they can’t see. For example, participants in one study (Newman, Wolff, & Hearst, 1980) played a game in which they were shown a set of trigrams, which are three-letter combinations such as SXY, GTR, BCG, and EVX. On each trial, the experimenter pointed to one of the trigrams in the set and told the participants that this trigram was the special one. The participants’ job was to figure out what made the special trigram so special. For half the participants, the special trigram was always the one that contained the letter T, and participants in this condition needed to see about 34 sets of trigrams before they figured out that the presence of T was what made the trigram special. But for the other half of the participants, the special trigram was always the one that lacked the letter T. How many trials did it take before participants figured it out? In fact, they never figured it out! Experiments like this one suggest that people consider what they see, but rarely consider what they don’t.

Why is it important to consider unseen evidence?

No matter how well the contestants perform, Jeopardy host Alex Trebek always seems like the smartest guy on the stage. But would you still get that impression if contestants were given an opportunity to ask him a few questions?
Amanda Edwards/Getty Images

This tendency can cause us to draw all kinds of erroneous conclusions. Consider a study in which participants were randomly assigned to play one of two roles in a game (Ross, Amabile, & Steinmetz, 1977). The “quizmasters” were asked to make up a series of difficult questions, and the “contestants” were asked to answer them. If you give this a quick try, you will discover that it’s very easy to generate questions that you can answer but that most other people cannot. For example, think of the last city you visited. Now give someone the name of the hotel you stayed in and ask him or her what street it’s on. They’re not likely to know.

46

So participants who were cast in the role of quizmaster naturally asked lots of clever-sounding questions, and participants who were cast in the role of contestant naturally gave lots of wrong answers. Now comes the interesting part. Quizmasters and contestants played this game while another participant—the observer—watched. After the game was over, the observer was asked to make some guesses about what the players were like in their everyday lives. The results were clear: Observers consistently concluded that the quizmaster was a more knowledgeable person than the contestant. Observers saw the quizmaster asking sharp questions and saw the contestant saying, “Um…er…uh…I don’t know,” and observers considered this evidence. What they failed to consider was the evidence they did not see. Specifically, they failed to consider what would have happened if the person who had been assigned to play the role of quizmaster had instead been assigned to play the role of contestant, and vice versa. If that had happened, then surely the contestant would have been the one asking clever questions, and the quizmaster would have been the one struggling to answer them. Bottom line? The first step in critical thinking is to doubt what you do see, and the second step is to consider what you don’t.

DATA VISUALIZATION

Does SAT Performance Correlate to Family Income and Education Level?
www.macmillanhighered.com/launchpad/schacterbrief3e

The Skeptical Stance

Winston Churchill once said that democracy is the worst form of government, except for all the others. Similarly, science is not an infallible method for learning about the world; it’s just a whole lot less fallible than the other methods. Science is a human enterprise, and humans make mistakes. They see what they expect to see, and they rarely consider what they can’t see at all.

Nick Downes/Cartoonstock.com

What makes science different from most other human enterprises is that scientists actively seek to discover and remedy these biases. Scientists are constantly striving to make their observations more accurate and their reasoning more rigorous, and they invite anyone and everyone to examine their evidence and challenge their conclusions. As such, science is the ultimate democracy—one of the only sports in which the lowliest nobody can triumph over the most celebrated someone. When an unknown patent clerk named Albert Einstein challenged the greatest physicists of his day, he didn’t have a famous name, a fancy degree, powerful friends, or a fat wallet. He just had evidence. And he prevailed for one reason: His evidence was right.

So think of the remaining chapters in this book as a report from the field—a description of the work that psychological scientists have done as they stumble toward knowledge. These chapters tell the story of the men and women who have put their faith in the scientific method and used it to pry loose small pieces of the truth about who we are, how we work, and what we are all doing here together on the third stone from the sun. Read it with interest but also with skepticism. Think critically about what you read here—and everywhere else.

47

SUMMARY QUIZ [2.3]

Question 2.7

1. When we observe a natural correlation, what keeps us from concluding that one variable is the cause and the other is the effect?
  1. the third-variable problem
  2. random assignment
  3. random sampling
  4. statistical significance

a.

Question 2.8

2. A researcher administers a questionnaire concerning attitudes toward global warming to people of both genders and of all ages who live all across the country. The dependent variable in the study is the participant’s _____________.
  1. age
  2. gender
  3. attitudes toward global warming
  4. geographic location

c.

Question 2.9

3. The characteristic of an experiment that allows conclusions about causal relationships to be drawn is called
  1. external validity.
  2. internal validity.
  3. generalization.
  4. self-selection.

b.

Question 2.10

4. When people find evidence that confirms their beliefs, they often
  1. tend to stop looking their beliefs.
  2. seek additional evidence that disconfirms them.
  3. consider what they cannot see.
  4. think critically about it.

a.