Computers are great for statistics. They can crunch numbers and do math faster and more accurately than humans can. But even a NASA supercomputer couldn’t do what we are about to do: use common sense to explain the results of a single-sample t test in plain English. Interpretation is the most human part of statistics.

To some degree, interpretation is a subjective process. The researcher takes objective facts—like the means and the test statistic—and explains them in his or her own words. How one researcher interprets the results of a hypothesis test might differ from how another researcher interprets them. Reasonable people can disagree. But, there are guidelines that researchers need to follow. An interpretation needs to be supported by facts. One person may perceive a glass of water as half full and another as half empty, but they should agree that it contains roughly equal volumes of water and air.

229

Interpreting the results of a hypothesis test starts with questions to be answered. The answers will provide the material to be used in a written interpretation. For a single-sample t test, our three questions are:

The questions are sequential. Each one gives new information that is useful in understanding the results from a different perspective. Answering only the first question, as was done in the last chapter, will provide enough information for completing a basic interpretation. However, answering all three questions leads to a more nuanced understanding of the results and a better interpretation.

Let’s follow Dr. Farshad as she interprets the results of her study, question by question. But, first, let’s review what she did. Dr. Farshad wondered if adults with ADHD differed in reaction time from the general population. To test this, she obtained a random sample, from her state, of 141 adults who had been diagnosed with ADHD. She had each participant take a reaction time test and found the sample mean and sample standard deviation (M = 220 msec, s = 27 msec). She planned to use a single-sample t test to compare the sample mean to the population mean for adult Americans (μ = 200 msec).

Dr. Farshad’s hypotheses for the single-sample t test were nondirectional as she was testing whether adults with ADHD had faster or slower reaction times than the general population. She was willing to make a Type I error no more than 5% of the time, she was doing a two-tailed test, and she had 140 degrees of freedom, so the critical value of t was ±1.977. The first step in calculating t was to find the estimated standard error of the mean (s_M = 2.21) and she used that in order to go on and find t = 8.81.

Step 6 Interpret the Results

Was the Null Hypothesis Rejected?

In Dr. Farshad’s study, the t value was calculated as 8.81. Now it is time to decide whether the null hypothesis was rejected or not. To do so, Dr. Farshad puts the t value, 8.81, into the decision rule that was generated for the critical value of t, ±1.977, in Step 4. Which of the following statements is true?

The second part of the first statement is true: 8.81 is greater than or equal to 1.977. The observed value of t, 8.81, falls in the rare zone of the sampling distribution, so the null hypothesis is rejected (see Figure 7.6). She can conclude that there is a statistically significant difference between the sample mean of the adults with ADHD and the general population mean.

Rejecting the null hypothesis means that Dr. Farshad has to accept the alternative hypothesis and conclude that the mean reaction time for adults with ADHD is different from the mean reaction time for the general population. Now she needs to determine the direction of the difference. By comparing the sample mean (220) to the population mean (200), she can conclude that adults with ADHD take a longer time to react and so have a slower reaction time than adults in general.

Dr. Farshad should also report the results in APA format. APA format provides five pieces of information: (1) what test was done, (2) the number of cases, (3) the value of the test statistic, (4) the alpha level used, and (5) whether the null hypothesis was or wasn’t rejected.

230

In APA format, the results would be t(140) = 8.81, p < .05:

If Dr. Farshad stopped after answering only the first interpretation question, this is what she could write for an interpretation:

There is a statistically significant difference between the mean reaction time of adults in Illinois with ADHD and the reaction time found in the general U.S. population, t(140) = 8.81, p < .05. Adults with ADHD (M = 220 msec) have a slower mean reaction time than does the general public (μ = 200).

How Big Is the Effect?

231

All we know so far is that there is a statistically significant difference, such that we can conclude that adults with ADHD in Illinois have slower reaction times than the general American adult population does. We know there’s a 20-msec difference, but we are hard-pressed to know how much of a difference that really is.

Thus, the next question to address when interpreting the results is effect size, how large the impact of the explanatory variable is on the outcome variable. With the reaction-time study, the explanatory variable is ADHD status (adults with ADHD vs. the general population) and the outcome variable is reaction time. Asking what the effect size is, is asking how much impact ADHD has on a person’s reaction time.

In this section, we cover two ways to measure effect size: Cohen’s d and r². Both can be used to determine if the effect is small, medium, or large. Both standardize the effect size, so outcome variables measured on different metrics can be compared. A 2-point change in GPA would be huge, while a 2-point change in total SAT score would be trivial. Both Cohen’s d and r² take the unit of measurement into account. And, finally, both will be used as measures of effect size for other statistical tests in other chapters.

Cohen’s d

The formula for Cohen’s d is shown in Equation 7.3. It takes the difference between the two means and standardizes it by dividing it by the sample standard deviation. Thus, d is like a z score—it is a standard score that allows different effects measured by different variables in different studies to be expressed—and compared—with a common unit of measurement.

To calculate d, Dr. Farshad needs to know the sample mean, the population mean, and the sample standard deviation. For the reaction-time data, M = 220, μ = 200, and s = 27. Here are her calculations for d:

232

Cohen’s d = 0.74 for the ADHD reaction-time study. Follow APA format and use two decimal places when reporting d. And, because d values can be greater than 1, values of d from –0.99 to 0.99 get zeros before the decimal point.

In terms of the size of the effect, it doesn’t matter whether d is positive or negative. A d value of 0.74 indicates that the two means are 0.74 standardized units apart. This is an equally strong effect whether it is 0.74 or –0.74. But the sign associated with d is important for knowing the direction of the difference, so make sure to keep it straight.

Here’s how Cohen’s d works:

Cohen (1988), the developer of d, has offered standards for what small, medium, and large effect sizes are in the social and behavioral sciences. Cohen’s d values for these are shown in Table 7.3. In Figure 7.7, the different effect sizes are illustrated by comparing the IQ of a control group (M = 100) to the IQ of an experimental group that is smarter by the amount of a small effect (3 IQ points), a medium effect (7.5 IQ points), or a large effect (12 IQ points).

Cohen describes a small effect as a d of around 0.20. This occurs when there is a small difference between means, a difference in performance that would not be readily apparent in casual observation. Look at the top panel in Figure 7.7. That’s what a small effect size looks like. Though the two groups differ by 3 points on intelligence (M = 100 vs. M = 103), it wouldn’t be obvious that one group is smarter than the other without using a sensitive measure like an IQ test (Cohen, 1988).

Another way to visualize the size of the effect is to consider how much overlap exists between the two groups. If the curve for one group fits exactly over the curve for the other, then the two groups are exactly the same and there is no effect. As the amount of overlap decreases, the effect size increases. For the small effect in Figure 7.7, about 85% of the total area overlaps.

A medium effect size, a Cohen’s d of around 0.50, is large enough to be observable, according to Cohen. Look at the middle panel in Figure 7.7. One group has a mean IQ of 100, and the other group has a mean IQ of 107.5, a 7.5 IQ point difference. Notice the amount of differentiation between groups for a medium effect size. In this example, about two-thirds of the total area overlaps, a decrease from the 85% overlap seen with a small effect size.

233

A large effect size is a Cohen’s d of 0.80 or larger. This is shown in the bottom panel of Figure 7.7. One group has a mean IQ of 100, and the other group has a mean IQ of 112, a 12 IQ point difference. Here, there is a lot of differentiation between the two groups with only about half of the area overlapping. The increased differentiation can be seen in the large section of the control group that has IQs lower than the experimental group and the large section of the experimental group that has IQs higher than the control group.

The Cohen’s d calculated for the reaction-time study by Dr. Farshad was 0.74, which means it is a medium to large effect. If she stopped her interpretation after calculating this effect size, she could add the following sentence to her interpretation: “The effect of ADHD status on reaction time falls in the medium to large range, suggesting that the slower mean reaction time associated with having ADHD may impair performance.”

234

r Squared

The other commonly used measure of effect size is r². Its formal name is coefficient of determination, but everyone calls it r squared. r², like d, tells how much impact the explanatory variable has on the outcome variable.

This formula says that r² is calculated as the squared t value divided by the sum of the squared t value plus the degrees of freedom for the t value. Then, to turn it into a percentage, the ratio is multiplied by 100. r² can range from 0% to 100%. These calculations reveal the percentage of variability in the outcome scores that is accounted for (or predicted) by the explanatory variable. For the ADHD data, these calculations would lead to the conclusion that r² = 36%:

What does r² tell us? Imagine that someone took a large sample of people and timed them running a mile. There would be a lot of variability in time, with some runners taking 5 minutes and others 20 minutes or more. What are some factors that influence running speed? Certainly, physical fitness and age play important roles, as do physical health and weight. Which of these four factors has the most influence on running speed? That could be determined by calculating r² for each variable. The factor that has the largest r², the one that explains the largest percentage of variability in time, has the most influence.

The question addressed by r² in the reaction-time study is how much of the variability in reaction time is accounted for by the explanatory variable (ADHD status) and how much is left unaccounted for:

Cohen (1988), who provided standards for d values, also provides standards for r²:

By Cohen’s standards, an r² of 36%, as is the case in our ADHD study, is a large effect. This means that as far as explanatory variables in the social and behavioral sciences go, this one accounts for a lot of the variability in the outcome variable. Figure 7.8 is a visual demonstration of what explaining 36% of the variability means. Note that although a majority of the variability, 64%, in reaction time remains unaccounted for, this is still considered a large effect. Here is what Dr. Farshad could add to her interpretation now that r² is known: “Having ADHD has a large effect on one’s reaction time. In the present study, knowing ADHD status explains more than a third of the variability in reaction time.”

One important thing to note is that even though Dr. Farshad went to the trouble of calculating both d and r², it is not appropriate to report both. These two provide overlapping information and it does not make a case stronger to say that the effect was a really strong one, because both d and r² are large. This would be like testing boiling water and reporting that it was really, really boiling because it registered 212 degrees on a Fahrenheit thermometer and 100 degrees on a Celsius thermometer.

Here’s a heads up for future chapters and for reading results sections in psychology articles—there are other measures of effect size that are similar to r². Both η² (eta squared) and ω² (omega squared) provide the same information, how much of the variability in the outcome variable is explained by the explanatory variable.

236

How Wide Is the Confidence Interval?

In Chapter 5, we encountered confidence intervals for the first time. There, they were used to take a sample mean, M, and estimate a range of values that was likely to capture the population mean, μ. Now, for the single-sample t test, the confidence interval will be used to calculate the range for how big the difference is between two population means. Two populations? Dr. Farshad’s confidence interval, in the ADHD reaction-time study, will tell how large (or small) the mean difference in reaction time might be between the population of adults with ADHD and the general population of Americans.

The difference between population means can be thought of as an effect size. If the distance between two population means is small, then the size of the effect (i.e., the impact of ADHD on reaction time) is not great. If the two population means are far apart, then the size of the effect is large.

Equation 7.5 is the formula for calculating a confidence interval for the difference between two population means. It calculates the most commonly used confidence interval, the 95% confidence interval.

Here are all the numbers Dr. Farshad needs to calculate the 95% confidence interval:

237

Applying Equation 7.5, she would calculate

95% CIμ_Diff = (M – μ) ± (t_cv × s_M)

= (220 – 200) ± (1.977 × 2.27)

= 20.0000 ± 4.4878

= from 15.5122 to 24.4878

= from 15.51 to 24.49

The 95% confidence interval for the difference between population means ranges from 15.51 msec to 24.49 msec. In APA format, this confidence interval would be reported as 95% CI [15.51, 24.49].

What information does a confidence interval provide? The technical definition is that if one drew sample after sample and calculated a 95% confidence interval for each one, 95% of the confidence intervals would capture the population value. But, that’s not very useful as an interpretative statement. Two confidence interval experts have offered their thoughts on interpreting confidence intervals (Cumming and Finch, 2005). With a 95% confidence interval, one interpretation is that a researcher can be 95% confident that the population value falls within the interval. Another way of saying this is that the confidence interval gives a range of plausible values for the population value. That means it is possible, but unlikely, that the population value falls outside of the confidence interval. A final implication is that the ends of the confidence interval are reasonable estimates of how large or how small the population value is. The end of the confidence interval closer to zero can be taken as an estimate of how small the population value might be, while the end of the confidence interval further from zero estimates how large the population value might be.

Dr. Farshad’s confidence interval tells her that there’s a 95% chance that the difference between the two population means falls somewhere in the interval from 15.51 msec to 24.49 msec. It means that the best prediction of how much slower reaction times are for the population of adults with ADHD than for the general population of Americans ranges from 15.51 msec slower to 24.49 msec slower (Figure 7.9).

Our interpretation of confidence intervals will focus on three aspects: (1) whether the value of 0 falls within the confidence interval; (2) how close the confidence interval comes to zero or how far from zero it goes, and (3) how wide the confidence interval is. These three aspects are explained below and summarized in Table 7.4.

238

Applying Equation 7.6 to her data, Dr. Farshad calculates 24.49 – 15.51 = 8.98. The confidence interval is almost 9 msec wide. Is this wide or narrow?

Evaluating the width of a confidence interval often requires expertise. It takes a reaction-time researcher like Dr. Farshad to tell us whether a 9-msec range for a confidence interval is a narrow range and sufficiently precise. Here, given the width of the confidence interval, she feels little need to replicate the study with a larger sample size to narrow the confidence interval.

Dr. Farshad has completed her study using a single-sample t test to see if adults with ADHD had a reaction time that differed from that for the general population. By addressing the three questions in Table 7.5, Dr. Farshad has gathered all the pieces she needs to write an interpretation. There are four points she addresses in her interpretation:

240

Here is Dr. Farshad’s interpretation:

A study compared the reaction time of a random sample of Illinois adults who had been diagnosed with ADHD (M = 220 msec) to the known reaction time for the American population (μ = 200 msec). The reaction time of adults with ADHD was statistically significantly slower than the reaction time found in the general population [t(140) = 8.81, p < .05]. The size of the difference in the larger population probably ranges on this task from a 16- to a 24-msec decrement in performance. This is not a small difference—these results suggest that ADHD in adults is associated with a medium to large level of impairment on tasks that require fast reactions. If one were to replicate this study, it would be advisable to obtain a broader sample of adults with ADHD, not just limiting it to one state. This would increase the generalizability of the results.