Research designs are plans for collecting scientific evidence. Now let’s look at the evidence that is collected.

When psychologists gather scientific evidence, they call it “data.” Data are any type of information obtained in a scientific study. There are two kinds: quantitative and qualitative.

Quantitative data are numerical. In quantitative research methods, participants’ responses are described in terms of numbers. A psychologist who computes people’s scores on an intelligence test, or records heart rate when workers with a tight job deadline are stressed, or times the number of seconds it takes people to perform a task, or counts the number of men who call a female experimenter (from our opening story) is collecting quantitative data. Most studies in psychology employ quantitative methods.

Qualitative data are sources of scientific information that are not converted into numbers. Qualitative research methods, then, are any of a wide range of methods in which researchers observe, record, and summarize behavior using words rather than numbers.

Let’s begin our coverage of data in psychology with quantitative data.

52

One of the biggest surprises of introductory psychology is all the numbers involved. Before taking the course, you might expect that words would suffice. When you contemplate the thoughts in your mind, you mostly encounter words, not numbers. When you experience complex, “mixed” emotions—the sort of feeling you might have if a boyfriend you want to stop seeing calls you to break up—you struggle to put your feelings into words, not numbers. But psychologists studying thoughts and emotions use numbers aplenty: graphs, charts, numerical tables, and the statistical analyses that produce them. Where do all these numbers come from?

MEASUREMENT. Psychologists get their numbers through a process called measurement, which is any technique that assigns numbers to information about objects or events (Stevens, 1948). (Later we’ll present the specialized scientific equipment used to get this information, in the section “Obtaining Scientific Evidence.” Here, we focus on the process of measurement itself.)

When it comes to physical properties, measurement is familiar. You measure length with a ruler and temperature with a thermometer. In doing so, you assign numbers (in centimeters and degrees, respectively) to physical properties (the distance from one end of a pencil to the other; the warmth of a room).

But what about psychological properties? People’s thoughts, feelings, abilities, attitudes, and personality characteristics don’t have length, width, or mass. So how can we measure them? The key step is to operationally define the psychological characteristics.

An operational definition specifies a procedure through which a property can be measured. The result of the procedure is, by definition, a measurement of the property. Here are examples of operational definitions for three psychological properties: (1) intelligence, (2) self-esteem, and (3) fearful reactions— in particular, among infants:

53

Sometimes researchers employ different operational definitions of the same property. One may operationally define fear, for example, in terms of behavioral reactions, whereas another might operationally define it with a questionnaire that asks people to describe their tendency to become fearful. These differences can cause substantial problems when one tries to interpret research findings (Kagan, 1988). The different procedures might produce conflicting results. Consider what would happen in the case of studying fear if some people who, behaviorally, experience a lot of fear describe themselves on the questionnaire as calm and fearless (Myers, 2010). (They might say this to impress others or to convince themselves that they are not so afraid.) Research using a behavioral definition would indicate that those people are “high in fear,” whereas research using a questionnaire-based definition would indicate that they are “low in fear.”

54

RELIABILITY AND VALIDITY. Once you decide on a measurement procedure, how do you know it is a good one? Measures should have two properties: reliability and validity.

It is possible for a measure to be reliable without being valid. Revisiting the 100-meter dash example, the measure itself might be reliable—that is, people’s times might be consistent from one running of the race to another. But no matter how reliable that information is, it still would not be a valid measure of intelligence.

THE ADVANTAGES OF NUMBERS. Measurement, then, is the answer to “Where are all these numbers coming from?” There’s another question, however: Why do we want numbers, or what are the advantages of quantitative data?

There are three advantages, involving comparison, conciseness, and precision.

55

THIS JUST IN

MEASUREMENT PRINCIPLES IN PRACTICE. Now let’s see how measurement works in practice. We’ll look at two examples, the first of which involves political attitudes.

Suppose you want to study attitudes about a political question: Should the government allow companies to drill for oil in wilderness areas? If you ask people to express their attitudes in words, you will get information that is neither concise nor precise (“Well, I don’t know, I guess I’m sort of against it, but the nation does need a lot of energy”). So you need numbers. To get them, take the two steps described earlier:

Researchers would ask precisely the same question to all participants, each of whom would respond using the same easy-to-interpret scale. Note that this procedure overcomes two problems in the dorm study you read about earlier—varied phrasing of the question may have affected the answers, and some responses were hard to interpret.

Our second example is a little more complex. Psychologists who study performance in mathematics may ask whether any two math problems—let’s call them Problem A and Problem B—differ systematically in the number of steps of thinking required to solve them. The variable of interest, then, is amount of thinking. How can one measure amount of thinking?

58

SUMMARIZING AND ANALYZING NUMBERS: STATISTICS. Suppose you were measuring a variable discussed earlier, attitudes about oil drilling in wilderness areas. After you’ve collected 100 people’s responses, someone asks, “What were people’s attitudes like?” There are two ways to answer. You could list all 100 responses, one by one: “The first person indicated a 5 on my 7-point scale. The second person indicated 3. The third person was another 5—hey, what a coincidence, just like the first person! And then the fourth person. …” But that method would be terribly inconvenient. The second, better option is to use statistics.

Statistics are mathematical procedures for summarizing sets of numbers. Some of the procedures, called descriptive statistics, describe what, in general, the numbers are like. Others, called inferential statistics, help scientists draw conclusions about the numbers (to conclude whether one group of numbers differs from a second group). This book’s Statistics Appendix presents descriptive and inferential statistics in detail. Here, we provide a brief overview.

When describing a set of numbers, one commonly wants to know what the numbers are like, on average. You can probably tell intuitively that answering the question “What were people’s attitudes like?” requires knowing the average attitude score. You can determine the average by computing the statistical mean, the value obtained by adding up all the scores and dividing by the number of scores that there were. (For our attitude example, you would add up the scores and divide by 100.)

A second descriptive statistic is the standard deviation, which describes the degree to which numbers vary (or “deviate”) from the mean. Consider two sets of numbers:

Set A: 1 1 2 2 4 6 6 7 7

Set B: 3 3 3 3 4 5 5 5 5

They have the same mean (4). But in Set A, the numbers vary from the mean to a greater degree than in Set B. The numbers in Set A therefore have a larger standard deviation.

After summarizing results with statistics such as the mean and standard deviation, psychologists often want to know whether the results differ from chance. The concept of differing from chance can be illustrated with a simple example. A flipped coin is expected to come up heads half the time and tails half the time. If you flip it 100 times, and it comes up heads 52 times instead of 50, you don’t think, “Unbelievable! Two more heads than the expected 50!” You know this small difference, 52 instead of 50, can occur just by chance. But now imagine that instead of 52 heads, you got 92. That would be bizarre! It would differ so much from what was expected—that is, so much from the chance result—that you would suspect the coin had been “rigged” to come up heads more than half the time.

Statistical procedures can determine whether observed outcomes differ from the outcome expected by chance. Observed outcomes that differ from what would be expected by chance are called statistically significant. In our example above, 92 heads out of 100 would be a statistically significant outcome, whereas 52 out of 100 would not. The inferential statistics used to determine whether results vary from chance are called significance tests. (Principles of statistical significance are detailed in our Statistics Appendix.)

A common application of significance tests is determining whether two or more groups of numbers—such as mean scores of different groups of people in an experiment—differ from one another. For example, back in Chapter 1, you read about a study in which men and women differed in their performance on a math test, in part due to stereotype threat. Before saying that the groups “differed,” the researchers conducted a significance test to find out whether the mean score in the different groups exceeded anything one would expect by chance alone.

59

Researchers usually analyze data from one study at a time. However, they sometimes summarize results from multiple studies. Meta-analysis is a statistical technique for combining results from multiple studies in order to identify overall patterns in the studies as a whole (Cooper, Hedges, & Valentine, 2009). So, in future chapters, when you read that researchers conducted a “meta-analysis,” it merely means that they summarized results from numerous prior studies.

LIMITATIONS OF QUANTITATIVE DATA. Quantitative data are the bread-and-butter of psychological science: the source of information that fuels most of the field’s growth. Yet some forms of quantitative data have limitations. Keep them in mind as you think critically about psychological research.

One shortcoming is that quantitative measures may fail to reveal information that could be detected in qualitative research (discussed below). In many quantitative studies, researchers construct a survey or questionnaire that is administered, in the same form, to all research participants. But participants might have important thoughts, feelings, and personal experiences not included in the survey. Suppose you wanted to survey college freshmen’s thoughts about factors that had “the biggest impact” on their past education. You might ask them to rate, on 7-point scales, factors such as the quality of their teachers, the support of parents, and the amount they studied. Thanks to the Internet, you could send your survey to people around the world. Some of them, though, might want to report information that you didn’t include in the survey. For instance, if asked about “the biggest impact” on their past education, students in Indonesia might want to say “the tsunami.” (A tsunami late in the year 2004 destroyed hundreds of Indonesian school buildings.) But your quantitative measure didn’t ask about that.

Another limitation is that numbers may fail to represent the complexity of some psychological characteristics. Consider people’s personalities. Many personality tests assign to individuals scores on personality traits such as being warm-hearted, organized, or a procrastinator (see Chapter 13). But when people describe themselves in words, their descriptions sometimes reveal personality characteristics that are more complex than the trait scores suggest. For example, in one study (Orom & Cervone, 2009), four people described themselves as follows:

60

No single “procrastination score” could, for example, describe the fourth person’s tendency to procrastinate, which varies from one situation to another. A single quantitative “procrastination score,” then, would not do a good job of describing that person. With such concerns in mind, some researchers opt for qualitative data.

Qualitative data, as we noted above, are sources of scientific information that are not converted into numbers. Words are the main type of qualitative data. Psychologists conducting qualitative research may ask participants to describe, in their own words, their experiences and life circumstances. They then summarize and interpret participants’ statements in words, rather than numbers.

The biggest source of qualitative information in psychology is interviews (Potter & Hepburn, 2005). Other information sources, such as observation of behavior in naturally occurring situations, also can provide qualitative research evidence. But interviews are most common. Let’s look at an example involving efforts to halt the spread of HIV/AIDS in sub-Saharan Africa.

HIV/AIDS efforts hold promise and have achieved much success already, yet they present a puzzle. Drugs can now stop the disease from progressing, and in sub-Saharan nations, they often are available for free. The puzzle is that a great many patients drop out of treatment; they skip the necessary trips to medical clinics to get their free drugs. Why would people fail to make these clinic visits, which are critical to their health?

A team of researchers knew the reasons could be complex; simple quantitative survey items might not uncover them. So they conducted a qualitative study. They interviewed patients in Nigeria, Tanzania, and Uganda who had missed treatment for three months or more (Ware et al., 2013). The patients discussed circumstances associated with their missed appointments, as well as their experiences at the medical clinics. Analysis of the study’s qualitative data indicated that patient dropout often resulted from “complex chains of events” (Ware et al., 2013, p. 6) involving both the challenges of life in sub-Saharan Africa and problems at the clinics. One man reported the following:

61

When you go to look for money for [a motorcycle taxi] you find you do not have it. So … you miss your appointment and go to clinic on another day, [and the provider] starts quarreling with you about not having come on the appointed day. And when you tell that person you got problems, he tells you, “You should spend the night on the road.” How can I spend the night on the road? Here I am, having failed to get money for taking me to the hospital and then I’m supposed to get money to spend the night somewhere and feed myself? These are some of the problems I have in going to the clinic.

—Ware et al. (2013, p. 6) *

* © 2013 Ware et al. (2013). Toward an understanding of disengagement from HIV treatment and care in sub-Saharan Africa: A qualitative study. PLoS Medicine, 10, e1001369, doi:10.1371/journal.pmed.1001469

Such complex, culturally situated reasons might never have been uncovered in a traditional quantitative study.

ADVANTAGES OF QUALITATIVE DATA. Three considerations motivate psychologists to obtain qualitative, rather than quantitative, data: the desire to (1) understand personal meaning, (2) reflect the storylike quality of lives, and (3) obtain evidence that is naturalistic. Let’s consider them.

The personal meaning of an event is its significance for the individual who is experiencing it. Personal meaning is hard to capture in numbers (Polkinghorne, 2005). Suppose a friend says, “I’m fed up with you.” You’re puzzled and want more information. But the information you want is not a number (“Exactly how fed up are you, on a scale from 1 to 7?”) Instead, you want to know exactly what the statement means—the reasons for, and behind, the statement. To understand the meaning of people’s statements, you generally need to let them speak in their own terms rather than handing them a fixed measurement scale (Kelly, 1955).

People’s understanding of their own lives often has a storylike structure— a narrative that unfolds over time (McAdams, 2006). Words, rather than numbers, are the best way to represent the information in life stories. Similarly, scholars note that life is like a play (Goffman, 1959; Scheibe, 2000). People in your life play different roles (friends, parents, teachers, romantic partners, etc.), and your behavior toward them, as well as theirs toward you, can be understood only by taking these roles into account. Comprehending the “theatrical” aspects of psychological experience may require qualitative methods that capture the meaning of behavior as it occurs in social contexts (Harré, 2000).

Naturalistic evidence is true-to-life information about the actual state of the world. Such evidence is what the psychologist desires, but sometimes quantitative methods make it hard to get. The procedures needed to obtain quantitative data may interfere with the natural flow of events. For example, suppose that, to measure the state of calm concentration that people sometimes experience while at work, you interrupt workers and ask them, “On a 7-point scale, to what extent are you experiencing a state of calm concentration right now?” Your intrusive quantitative procedure will interfere with people’s naturally occurring feelings. They might, for example, have been calmly concentrating until you interrupted them. Qualitative research strategies can avoid such intrusions. Qualitative researchers might, for example, observe ongoing events without interfering with them, or study preexisting written materials that provide information about people’s experiences (Denzin & Lincoln, 2011).

THREE TYPES OF QUALITATIVE STUDIES. Let’s look at three types of qualitative studies: (1) clinical case studies, (2) qualitative observational studies, and (3) qualitative community-participation studies.

62

A case study is a detailed analysis of one particular person or group of people; the person or group is the “case.” In psychology, a clinical case study is an analysis of someone receiving psychological therapy, which often is provided in a health center or clinic. Psychologists generally describe their cases in written summaries; clinical case studies, then, typically produce qualitative data.

A second type of research that can produce qualitative data is an observational study, in which researchers observe people’s behavior from a distance without interacting with them. A researcher might, for example, sit inconspicuously in a school playground and observe friendly and aggressive behavior among schoolchildren (Ostrov & Keating, 2004). Although the results of an observational study can include numerical data (such as number of aggressive incidents), researchers commonly summarize their observations in written reports, yielding qualitative data (Polkinghorne, 2005).

In one observational study, a researcher provided compelling evidence of differences between rich and poor neighborhoods (Zimbardo, 1970). He parked two similar cars in different locations—an inner-city neighborhood and a wealthy suburb—and then observed what happened. In the suburban community, there wasn’t much to observe; for five days, residents just passed the car by. But in the city, people began vandalizing the vehicle within 10 minutes of the study’s start. They removed everything of value within two days. The researcher speculated that the sense of anonymity that pervades large urban neighborhoods was the primary reason vandalism occurred. In suburban neighborhoods, by comparison, residents are less anonymous and thus share a sense of personal responsibility for community property. The researcher’s written report of his observations powerfully illustrated differences in social life from one neighborhood to another.

In an observational study, the psychologist does not interact with the research participants. However, in a different type of qualitative study, called a community-participation study, researchers collaborate with community residents to determine a study’s goals and research procedures (Allison & Rootman, 1996; Kelly, 2004). The psychologist’s aim is to understand community life and to help residents achieve social changes they desire. Because the residents are the experts on their community’s experiences and goals, the psychologist works closely with them in designing research.

The data in community-based studies generally are qualitative. Psychologists listen closely to residents’ reports about their lives, in order to understand the challenges they face in the community and how they cope with them. For example, in a study of how homeless mothers cope with the stresses of child care while living in poverty (Banyard, 1995), a psychologist interviewed 64 women living in emergency shelters in cities in the U.S. Midwest. An analysis of the qualitative data identified themes in the research participants’ statements. The most common theme was the need to confront problems directly—to cope by seeking solutions rather than just worrying about problems. “When one deal doesn’t work out,” one woman reported, when discussing her search for housing for herself and her child, “I swallow my pride and I call somebody else. … You just can’t give up” (Banyard, 1995, p. 881). These qualitative data provided a rich, emotionally meaningful understanding of homeless mothers’ lives that might not have been obtained if the psychologist had used numerical data instead.

63

DISADVANTAGES OF QUALITATIVE DATA. Despite their virtues, qualitative methods have drawbacks. Disadvantages lie in precisely those areas where numerical methods are advantageous.

With qualitative data, comparison is difficult. Suppose you read two case studies by a therapist and want to compare the people; you might ask, “Which of the two experienced more severe psychological distress?” Qualitative case studies provide no simple answer of the sort you would get from a quantitative measure.

Qualitative methods also are not concise. For example, an American psychologist once studied an individual’s personality qualitatively, by analyzing letters written by the person over a period of more than a decade (Allport, 1965). Interesting though it may have been, the resulting qualitative personality analysis was not concise: a book of more than 200 pages, describing only one individual. In addition, gathering such qualitative evidence is such a slow and labor-intensive process that qualitative methods inherently do not speed the path of scientific progress.

Because words are open to interpretation, qualitative methods also lack precision. Different speakers, with different personal and cultural backgrounds, may assign different meanings to the same word or phrase. In contrast, numbers are a “universal language” whose meaning is understood precisely.

Finally, qualitative data leave open the possibility of bias. When conducting a study, researchers commonly hope that the results are consistent with their own theoretical ideas. If the data are quantitative, researchers face unambiguous numerical facts that may disconfirm their hypotheses. But when the data are qualitative, they need to be interpreted, and the researcher’s hopes may bias the interpretation. One of the most famous sets of investigations in the history of psychology has been criticized on these grounds. Sigmund Freud (1923) developed a theory of personality (see Chapter 13) based entirely on qualitative case-study evidence. Critics note that the person who interpreted the case studies was Freud himself; the interpretations thus may have been biased by Freud’s desire to obtain evidence that supported his theory (Crews, 1998).

These disadvantages are substantial. As a result, most psychologists opt for quantitative rather than qualitative evidence. As we now turn to questions of research design, the studies we discuss will almost exclusively involve quantitative data.

64