2.4 The Experimental Method

An experiment is a research procedure in which a variable is manipulated and the manipulation’s effect on another variable is observed. In fact, most of us perform experiments throughout our lives without knowing that we are behaving so scientifically. Suppose that you go to a party on campus to celebrate the end of midterm exams. As you mix with people at the party, you begin to notice many of them becoming quiet and depressed. It seems the more you talk, the more subdued the other guests become. As the party falls apart before your eyes, you decide you have to do something, but what? Before you can eliminate the problem, you need to know what’s causing it.

experiment A research procedure in which a variable is manipulated and the effect of the manipulation is observed.

Your first hunch may be that something you’re doing is responsible. Perhaps your remarks about academic pressures have been upsetting everyone. You decide to change the topic to skiing in the mountains of Colorado and to watch for signs of depression in the next round of conversations. The problem seems to clear up; most people now smile and laugh as they chat with you. As a final check of your suspicions, you could go back to talking about school with the next several people you meet. Their dark and dismal reaction would probably convince you that your tendency to talk about school was indeed the cause of the problem.

41

You have just performed an experiment, testing your hypothesis about a causal relationship between your topic of conversation and the depressed mood of the people around you. You manipulated the variable that you suspected to be the cause (the topic) and then observed the effect of that manipulation on the other variable (the mood of the people around you). In scientific experiments, the manipulated variable is called the independent variable and the variable being observed is called the dependent variable.

independent variable The variable in an experiment that is manipulated to determine whether it has an effect on another variable.

dependent variable The variable in an experiment expected to change as the independent variable is manipulated.

Is animal companionship a form of therapy? A patient (right) and therapist (left) feed ring-tailed lemurs at Serengeti Park near Hodenhagen, Germany, as part of a monthly program called “Psychiatric Animal Days.” The program is based on the assumption that animals have a calming and therapeutic effect on people. As many as 400 kinds of therapies are currently used for psychological problems. An experimental design is needed to determine whether this or any other form of treatment causes clients to improve.

To examine the experimental method more fully, let’s consider a question that is often asked by clinicians (Toth et al., 2014; Cuijpers et al., 2013): “Does a particular therapy relieve the symptoms of a particular disorder?” Because this question is about a causal relationship, it can be answered only by an experiment (see Table 2-2). That is, experimenters must give the therapy in question to people who are suffering from a disorder and then observe whether they improve. Here the therapy is the independent variable, and psychological improvement is the dependent variable.

Table 2.2: table: 2-2Most Investigated Questions in Clinical Research

Most Common Correlational Questions

Are stress and onset of mental disorders related?

Is culture (or gender or race) generally linked to mental disorders?

Are income and mental disorders related?

Are social skills tied to mental disorders?

Is social support tied to mental disorders?

Are family conflict and mental disorders related?

Is treatment responsiveness tied to culture?

Which symptoms of a disorder appear together?

How common is a disorder in a particular population?

Most Common Causal Questions

Does factor X cause a disorder?

Is cause A more influential than cause B?

How does family communication and structure affect family members?

How does a disorder affect the quality of a person’s life?

Does treatment X alleviate a disorder?

Is treatment X more helpful than no treatment at all?

Is treatment A more helpful than treatment B?

Why does treatment X work?

Can an intervention prevent abnormal functioning?

If the true cause of changes in the dependent variable cannot be separated from other possible causes, then an experiment gives very little information. Thus, experimenters must try to eliminate all confounds from their studies—variables other than the independent variable that may also be affecting the dependent variable. When there are confounds in an experiment, they, rather than the independent variable, may be causing the observed changes.

confound In an experiment, a variable other than the independent variable that is also acting on the dependent variable.

42

For example, situational variables, such as the location of the therapy office (say, a quiet country setting) or soothing background music in the office, may have a therapeutic effect on participants in a therapy study. Or perhaps the participants are unusually motivated or have high expectations that the therapy will work, factors that thus account for their improvement. To guard against confounds, researchers should include three important features in their experiments—a control group, random assignment, and a blind design (see MediaSpeak below).

The Control Group

A control group is a group of research participants who are not exposed to the independent variable under investigation but whose experience is similar to that of the experimental group, the participants who are exposed to the independent variable. By comparing the two groups, an experimenter can better determine the effect of the independent variable.

control group In an experiment, a group of participants who are not exposed to the independent variable.

experimental group In an experiment, the participants who are exposed to the independent variable under investigation.

BETWEEN THE LINES

Recent Studies Conducted Online

Frequency of Adolescent Text Messaging

Attitudes Toward Adoption

Self-Presentation on Dating Websites

E-Mails to Improve Your Mood

Causes of Internet Pornography Usage

Do You Have a Positive Personality?

Are You a Cyberslacker?

You and Your Parents

Personality and Computer Game Use

(Underwood et al., 2012; Krantz, 2011)

To study the effectiveness of a particular therapy, for example, experimenters typically divide participants into two groups after obtaining their consent to participate in the experiment. The experimental group may come into an office and receive the therapy for an hour, while the control group may simply come into the office for an hour. If the experimenters find later that the people in the experimental group improve more than the people in the control group, they may conclude that the therapy was effective, above and beyond the effects of time, the office setting, and any other confounds. To guard against confounds, experimenters try to provide all participants, both control and experimental, with experiences that are identical in every way—except for the independent variable.

Of course, it is possible that the differences observed between an experimental group and control group have occurred simply by chance. Thus, as with correlational studies, investigators who conduct experiments must do a statistical analysis on their data and find out how likely it is that the observed differences are due to chance. If the likelihood is very low—less than 5 percent (p < .05)—the differences between the two groups are considered to be statistically significant, and the experimenter may conclude with some confidence that they are due to the independent variable. As a general rule, if the sample of participants is large, if the difference observed between groups is great, and if the range of scores within each group is small, the findings of an experiment are likely to be statistically significant.

An additional point is worth noting with regard to clinical treatment experiments. It is always important to distinguish between statistical significance and a notion called clinical significance. As you have just read, statistical significance indicates whether a participant’s improvement in functioning—large or small—occurred because of treatment. Clinical significance indicates whether the amount of improvement is meaningful in the individual’s life. Even if the moods of depressed participants improve because of treatment, the individuals may still be too unhappy to enjoy life. Thus, although experimenters can determine statistical significance, only individuals and their clinicians can fully evaluate clinical significance.

Random Assignment

BETWEEN THE LINES

In Their Words

“Every individual is the exception to the rule.”

C. G. Jung, 1921

Researchers must also watch out for differences in the makeup of the experimental and control groups since those differences may also confound a study’s results. In a therapy study, for example, the experimenter may unintentionally put wealthier participants in the experimental group and poorer ones in the control group. This difference, rather than their therapy, may be the cause of the greater improvement later found among the experimental participants. To reduce the effects of preexisting differences, experimenters typically use random assignment. This is the general term for any selection procedure that ensures that every participant in the experiment is as likely to be placed in one group as the other (Jackson, 2012; Remler & Van Ryzin, 2011). Researchers might, for example, select people by flipping a coin or picking names out of a hat.

random assignment A selection procedure that ensures that participants are randomly placed either in the control group or in the experimental group.

43

MediaSpeak

Flawed Study, Gigantic Impact

By David DiSalvo, Forbes, May 19, 2012

In 2001, Dr. Robert L. Spitzer, psychiatrist and professor emeritus of Columbia University, presented a paper at a meeting of the American Psychiatric Association about something called “reparative therapy” [also known as “conversion therapy”] for gay men and women. By undergoing reparative therapy, the paper claimed, gay men and women could change their sexual orientation. Spitzer had interviewed 200 allegedly former-homosexual men and women that he claimed had shown varying degrees of such change; all of the participants provided Spitzer with self reports of their experience with the therapy.

Protesting reparative therapy Protestors from a gay rights group in Hong Kong hold up a banner outside a social welfare department in 2011 to protest the department’s endorsement of reparative therapy.

Spitzer, now 79 years old, was no stranger to the controversy surrounding his chosen subject. Thirty years earlier, he had played a leading role in removing homosexuality from the list of mental disorders in the association’s diagnostic manual [DSM-III]. Clearly, his interest in the topic was more than a passing academic curiosity….

Fast forward to 2012, and Spitzer is of quite a different mind. Last month he told a reporter with The American Prospect that he regretted the 2001 study and the effect it had on the gay community, and that he owed the community an apology. And this month he sent a letter to the Archives of Sexual Behavior, which published his work in 2003, asking that the journal retract his paper.

Why might an outstanding and highly regarded researcher have made such errors in the conduct and interpretation of this study?

Spitzer’s mission to clean the slate is commendable, but the effects of his work have been coursing through the homosexual community like acid since it made headlines a decade ago. His study was seized upon by anti-homosexual activists and therapists who held up Spitzer’s paper as proof that they could “cure” patients of their sexual orientation.

Spitzer didn’t invent reparative therapy, and he isn’t the only researcher to have conducted studies claiming that it works, but as an influential psychiatrist from a prestigious university, his words carried a lot of weight.

In his recantation of the study, he says that it contained at least two fatal flaws: the self reports from those he surveyed were not verifiable, and he didn’t include a control group of men and women who didn’t undergo the therapy for comparison. Self reports are notoriously unreliable … Lacking a control group is a fundamental no-no in social science research across the board. The conclusion is inescapable—Spitzer’s study was simply bad science….

Reading the study now, I’m sure Spitzer is embarrassed by its flaws. Not only did he rely on self reports, but he conducted the participant interviews by phone, which escalates unreliability to the doesn’t-pass-the-laugh-test level. By phone, researchers aren’t able to evaluate essential non-verbal cues that might cast doubts on verbal responses. Phone interviews, along with written interviews, carry too much guesswork baggage to be valuable in a scientific study, and Spitzer certainly knew that.

The object lesson worth drawing from this story is that just one instance of bad science given the blessing of recognized experts can lead to years of damaging lies that snowball out of control. Spitzer cannot be held solely responsible for what happened after his paper was published, but he’d probably agree now that the study should never have been presented in the first place. At the very least, his example may help prevent future episodes of the same.

May 19, 2012, “How One Flawed Study Spawned a Decade of Lies” by David DiSalvo. From Forbes, 5/19/2012 © 2012 Forbes LLC. All rights reserved. Used by permission and protected by the copyright laws of the United States. The printing, copying, redistribution, or retransmission of this content without express written permission is prohibited.

44

Blind Design

A final confound problem is bias. Participants may bias an experiment’s results by trying to please or help the experimenter (Goodwin & Goodwin, 2012). In a therapy experiment, for example, if those participants who receive the treatment know the purpose of the study and which group they are in, they might actually work harder to feel better or fulfill the experimenter’s expectations. If so, subject, or participant, bias rather than therapy could be causing their improvement.

To avoid this bias, experimenters can prevent participants from finding out which group they are in. This experimental strategy is called a blind design because the individuals are blind as to their assigned group. In a therapy study, for example, control participants could be given a placebo (Latin for “I shall please”), something that looks or tastes like real therapy but has none of its key ingredients. This “imitation” therapy is called placebo therapy. If the experimental (true therapy) participants then improve more than the control (placebo therapy) participants, experimenters have more confidence that the true therapy has caused their improvement.

Why might sugar pills or other kinds of placebo treatments help some people feel better?

blind design An experiment in which participants do not know whether they are in the experimental or the control condition.

placebo therapy A pretend treatment that the participant in an experiment believes to be genuine.

An experiment may also be confounded by experimenter bias—that is, experimenters may have expectations that they unintentionally transmit to the participants in their studies (see InfoCentral in the next section). In a drug therapy study, for example, the experimenter might smile and act confident while providing real medications to the experimental participants but frown and appear hesitant while offering placebo drugs to the control participants. This kind of bias is sometimes referred to as the Rosenthal effect, after the psychologist who first identified it (Rosenthal, 1966). Experimenters can eliminate their own bias by arranging to be blind themselves. In a drug therapy study, for example, an aide could make sure that the real medication and the placebo drug look identical. The experimenter could then administer treatment without knowing which participants were receiving true medications and which were receiving false medications.

While either the participants or the experimenter may be kept blind in an experiment, it is best that both be blind—a research strategy called a double-blind design. In fact, most medication experiments now use double-blind designs to test promising drugs (Pratley, Fleck, & Wilson, 2014; Wender et al., 2011). Many experimenters also arrange for judges to assess the patients’ improvement independently, and the judges, too, are blind to group assignments. This strategy is called a triple-blind design (de Paula et al., 2013).

double-blind design Experimental procedure in which neither the participant nor the experimenter knows whether the participant has received the experimental treatment or a placebo.