Think Like a Scientist
Susan A. Nolan, Seton Hall University
Sandra E. Hockenbury
With the assistance of Scott Cohn, Western State Colorado University
Fram, Alan. (2007). That’s the spirit: One-third of people believe in ghosts—and some report seeing one. Associated Press Archive, Record No. d8SGT3CG0. Associated Press, Washington, DC.
French, Chris. (2012). Halloween challenge: Psychics submit their powers to a scientific trial. The Guardian. Retrieved from http://www.theguardian.com/science/2012/oct/31/halloween-challenge-psychics-scientific-trial
Goldsmiths, University of London. (2012). Psychics fail ‘Halloween Challenge’ at Goldsmiths [Press release]. Retrieved from http://www.gold.ac.uk/news/pressrelease/?releaseID=972
Hyman, Ray. (2010). Meta-analysis that conceals more than it reveals: Comment on Storm et al. (2010). Psychological Bulletin, 136, 486-490.
Rouder, Jeffrey N.; Morey, Richard D.; & Province, Jordan M. (2013). A Bayes factor meta-analysis of recent extrasensory perception experiments: Comment on Storm, Tressoldi, and Di Risio (2010). Psychological Bulletin, 139, 241–247.
Storm, Lance; Tressoldi, Patrizio E.; & Di Risio, Lorenzo. (2010). Meta-analysis of free-response studies, 1992–2008: Assessing the noise reduction model in parapsychology. Psychological Bulletin, 136, 471–485.
What is Think Like a Scientist?
Think Like a Scientist is a digital activity designed to help you develop your scientific thinking skills. Each activity places you in a different, real-world scenario, asking you to think critically about a specific claim.
Can instructors track your progress inThink Like a Scientist?
Scores from the five-question assessments at the end of each activity can be reported to your instructor. To ensure your privacy while participating in non-assessment features, which can include pseudoscientific quizzes or games, no other student response is saved or reported.
How is Think Like a Scientist aligned with the APA Guidelines 2.0?
The American Psychological Association’s “Guidelines for the Undergraduate Psychology Major” provides a set of learning goals for students. Think Like a Scientist addresses several of these goals, although it is specifically designed to develop skills from APA Goal 2: Scientific Inquiry and Critical Thinking. “ESP” covers many outcomes, including:
This activity invites you to explore the existence of psychic abilities. First, you’ll identify the claim about extrasensory perception, or ESP. Then, you will explore evidence about the existence of ESP. You’ll also examine how your beliefs about ESP affect how you interpret research on this phenomenon. You will examine alternative explanations for why people believe in ESP despite a lack of evidence that it exists. Finally, you’ll consider the need to ask good questions about science, even when there is a high-quality source of available information, such as a peer-reviewed journal article.
Identify the Claim
Have you ever had a hunch that turned out to be true? Do you believe in clairvoyance, telepathy, or other forms of extrasensory perception (ESP)? As you learned in your textbook chapter on sensation and perception, ESP is the acquisition of information outside of the normal processes of sensation like seeing or hearing. It’s kind of like Bluetooth for your brain. About half of Americans believe in ESP (Fram, 2007). Are you one of them? Later, you can complete a short survey to find out.
If you are interested in paranormal phenomena, you may have looked for information about ESP online. Googling “Do I have ESP?” yields dozens of sites offering quizzes and exercises that claim to help you recognize or develop this “skill.” As an example, look at this mock Web site, which is similar to many “psychic phenomena” sites online. (You can click the image to enlarge.) On the next screen, you can take the quiz advertised on this site.
Imagine you are intrigued by the Web site you just viewed, and you want to take the “psychic inclinations” test. The mock quiz that follows was developed based on actual quizzes available online, and you can try it now. You must complete all 15 questions in this quiz in order to receive your “result” and advance to the next screen of this Think Like a Scientist activity.
Complete a short survey to find out if YOU have psychic inclinations! For each item, click either yes or no.
1. Do you feel energized by some people and exhausted after being around others?
2. Do you ever feel like plants have energy or a presence?
3. Have you ever dreamt about an event, or sequence of events, that later happened?
4. Do you ever feel the presence of people shortly after they die?
5. When meeting people for the first time, do you ever feel like you already know them from somewhere?
6. Do you ever feel like there is another presence helping you? For example, when struggling to arrive at a decision, does an answer ever ‘come to you’ out of the blue?
7. Are uncanny coincidences and serendipitous circumstances common in your life?
8. Do you communicate well with small children and animals?
9. Have you ever been alone and heard someone talking to you?
10. Do you ever make decisions to avoid certain situations because of a ‘gut feeling’ that something will go wrong?
11. Do you often think of people right before receiving a phone call or text from them?
12. Did you have imaginary friends as a child that you could see, but were invisible to others?
13. Have you ever seen or sensed an aura?
14. If you stare at someone from behind, do they often turn around to look at you?
15. When entering a space that you haven’t been in before, do you often get a feeling about the environment’s history that takes the form of an unexpected emotion or a physical sensation?
What does your score on the “Psychic Abilities Quiz” mean in terms of your own “psychic inclinations”? Do you believe some people taking this quiz will learn that they have the potential for actual psychic abilities? In your own words, explain what a quiz like this tells you. We’ll come back to your answer later.
Evaluate the Evidence
Although the general public is split about whether ESP is real, only a small number of researchers are actively investigating this topic. The vast majority of psychologists agree that there is no convincing scientific evidence that ESP exists (Hyman, 2010).
So if the evidence is overwhelming, are you wondering why there is any debate at all? Whether or not you asked that question may have something to do with your response to the quiz you just took. We’ll look at the evidence for and against the existence of ESP. But first, let’s examine how scientists investigate ESP.
The following small-scale example shows how ESP can be tested in the lab. Psychologists at the Anomalistic Psychology Research Unit, part of the University of London, put professional psychics Kim Whitton (shown below, at left) and Patricia Putt to the test. Whitton and Putt volunteered to provide psychic “readings” for five women (Goldsmiths, University of London, 2012). The entire test was conducted in silence. As you can see here, the women were separated by a screen from the psychics, who recorded their readings as written notes specific to each individual woman. The women were then asked to identify their own readings from among the five that Putt and Whitton had written. Although the psychics called the test “very fair and simple,” they both failed it. For Whitton, only one woman correctly guessed which reading described her—exactly what you’d expect by chance. And for Putt, not one woman guessed correctly.
Only a few psychology researchers study ESP. Researchers at the University of London regularly conduct experiments like the one just described, but have never found any support for claims about the existence of ESP. However, in a meta-analysis of 67 different studies, Australian psychologist Lance Storm and colleagues (2010) did find statistical support for the existence of ESP. In some of the studies that Storm and his colleagues examined, a “sender” was asked to telepathically broadcast information about an item to a “receiver” in a separate room. The “receiver” then attempted to choose that item from among a limited number of options.
The researchers did not explain how psychic abilities might work, usually an important part of a scientific write-up. And ESP cannot be explained by widely accepted principles of physics (Rouder & others, 2013). Nevertheless, findings from this meta-analysis (2010) provide evidence capable of convincing a skeptical but open-minded reader that ESP is real.
A few years later, U.S. psychologist Jeffrey Rouder and his colleagues performed their own meta-analysis, re-examining Storm and colleagues’ findings (2013). Focusing on studies that provided the most support for the existence of ESP, they found errors in many of them. Some studies, for example, did not use proper randomization procedures.
Ideally, researchers prevent any possibility of human interference—whether conscious or unconscious—by using a computer to determine relevant aspects of their experiment. For example, a computer can be used to randomly assign participants to groups or determine the order in which participants view items. However, in some studies included in the Storm team’s meta-analysis, researchers themselves made some decisions about the inclusion or order of items to be shown to the “sender.” Once human judgment enters the picture, the experiment is not truly random; there may be some pattern that is created, even unconsciously, that could change the results of a study.
Do you think it mattered that some studies used full randomization and others did not? Rouder and his colleagues compared these two different types of studies. They found that the experiments that used full computerized randomization showed many more results at or below the level of chance than the ones that did not use full computerized randomization. You can see this difference in the graph. When an outcome is at or below the level of chance, it does not provide support for the experiment’s hypothesis—in this case, the existence of ESP. So these studies provided less support for the existence of ESP than a meta-analysis including studies that used a manual randomization process.
Once scientists track down all of the research in a particular area, how do they choose which studies to include in a meta-analysis? They can’t just flip a coin. A good scientist comes up with careful guidelines and only uses studies that meet those guidelines.
Rouder and his colleagues questioned the selection process for the studies that Storm and his colleagues included in their meta-analysis. Rouder’s team cited examples of studies that they believed should not have been included, as well as ones they believed should have been included. In one case, a research team conducted two experiments that both seemed to meet the criteria to be included in the meta-analysis. Yet, Storm’s team only included one of them—the one that provided more support for the existence of psychic abilities. And this is just an examination of the published studies. What about the ones that have never seen the light of day?
It is well known that research that finds an effect—say, that ESP works or that there is a difference between two groups—is more likely to be published. Even editors of academic journals and of news sources like magazines are swayed by the excitement of a statistically significant finding. Imagine you are an editor considering which articles to include in the next issue of your popular magazine. In each pair, indicate which fictional headline draws your attention.
The way that research studies work is that they either: 1) provide evidence for something, as Storm’s team did with respect to ESP, or 2) fail to find evidence for something, as Rouder’s team did with respect to ESP. What this means is that Rouder’s team’s findings don’t allow us to say definitively that ESP doesn’t exist. However, their meta-analysis does show that the evidence favoring ESP is far weaker than Storm’s team claimed.
Rouder’s team concluded that the Storm team’s meta-analysis is “certainly not sufficient to sway an appropriately skeptical reader” (2013).
Consider Alternative Explanations
So, we now have a finding and what we might call a “non-finding”—study results that do not show support. How do you interpret the evidence? Your position in this “debate”—or even your thoughts on whether a debate really exists at all—may have something to do with your response to the quiz you completed earlier in this activity. After you took the quiz, you were asked what you thought the quiz could tell us. You said:
Most people have prior beliefs about the existence of ESP—they either believe it is possible or they believe it is impossible. Your beliefs likely guided whether you thought the quiz could tell you something or you just thought it was a gimmick.
If on the quiz you were open to the belief that you might have psychic powers, it is more likely that the Storm team’s meta-analysis results will remain compelling to you. It can be challenging to respond objectively to evidence that conflicts with our personal beliefs. Remember the psychics who were tested in the British study we discussed earlier? They interpreted the outcome of that test—their failure to show psychic abilities—in light of their own prior beliefs. After the test, psychic Kim Whitton said, “I know what I do is very real, it's easy for me. I'm glad one of the [women] could recognise so many details about herself” (Goldsmiths, University of London, 2012). Remember, one out of five guessing her reading correctly is exactly what you’d expect by chance. That’s like rolling a die repeatedly, focusing on the single time you roll a six, and saying “See how good I am at rolling a six?” But Whitton was not swayed by this evidence.
But what if you were skeptical of what it meant to score a five or higher on the quiz you took earlier? Did you recognize that some items on the quiz are linked to explainable phenomena like déjà vu or coincidence? If so, then you will likely also accept that this weakened support serves to discount the existence of ESP. For the half of all Americans who don’t believe in ESP, the results from Rouder and colleagues (2013) suggest that some form of systematic error may be at the root of studies purporting to show ESP.
Consider the Source of the Research or Claim
Thinking like a scientist also involves thoughtful consideration of the source of information. In this case both sources are excellent—peer-reviewed articles published in scholarly journals. This means that both Storm’s team (2010) and Rouder’s (2013) permitted their research to be reviewed by a panel of their peers prior to publication. Further, both teams published detailed explanations of the methods used and the results obtained rather than simply a summary of their findings. However, even peer-reviewed studies published in premier research journals are not beyond scrutiny.
Earlier we talked about how your prior beliefs affected what you think about scientific research. What about the scientists’ beliefs? Could they have played some role in the outcome of the meta-analyses? For example, Rouder and his colleagues pointed out systematic errors in Storm and colleagues’ meta-analysis. Imagine that Storm and his colleagues were open to the existence of ESP. Might their prior beliefs have unconsciously led to some of the choices they made about which studies to include?
Thinking like a scientist means being aware of your own beliefs and possible biases—whether you’re reading about science or conducting an experiment. Wherever your personal beliefs lie in any scientific debate, it’s always important to think like a scientist to ensure that you weigh equally the findings that support your beliefs as well as the findings that don’t.
If, after learning about the evidence, you believe you could still have psychic abilities, here’s a challenge. James Randi (pictured below), the magician known as “The Amazing Randi” and a self-described “scientific skeptic,” has funded an educational foundation that issued “The Million Dollar Challenge” (http://web.randi.org/the-million-dollar-challenge.html). If you can demonstrate that you are psychic, while following the foundation’s scientific protocols, you can win $1,000,000. No joke! This offer has been on the table since 1964, and hundreds have applied. So far, no one has won.