304

Interpreting a paired-samples t test starts by addressing the same questions that were used in interpreting the single-sample t test and the independent-samples t test:

And the interpretation ends the same way as well, with a written statement that covers four points:

Before Dr. Keim starts the interpretation, let’s review her study on the effect of humidity on perceived temperature. She used six participants in a within-subjects design, where each person was tested twice in a 76° room, once at low humidity and once at high humidity. The mean perceived temperature in the low-humidity condition was 75.00°F (s = 4.34°F) and 82.50°F (s = 4.93°F) in the high-humidity condition. The mean difference score was 7.50 (s_D = 2.07°F). For a two-tailed test with α = .05 and df = 5, t_cv was ±2.571°F. Dr. Keim calculated = 0.85°F and t = 8.82°F.

This first interpretation question can be answered by plugging the observed value of the test statistic, t = 8.82, into the decision rule Dr. Keim formulated in Step 4:

The first statement is true because 8.82 is greater than or equal to 2.571. This can be seen in Figure 9.5, where it is clear that the value of the test statistic falls in the rare zone. Dr. Keim has rejected the null hypothesis.

Rejecting the null hypothesis that the two population means are the same leads to accepting the alternative hypothesis that the two population means are different. Dr. Keim can say that a statistically significant difference exists between the perceived temperature in low humidity vs. high humidity.

305

The Direction of the Difference

Reporting that there is a difference is true, but it is not very useful. The logical follow-up question is, “What is the direction of the difference?” By examining the two sample means, 75.00° for the low-humidity condition vs. 82.50° for the high-humidity condition, Dr. Keim can conclude that 76° feels significantly hotter with high humidity than with low humidity.

A researcher only needs to worry about the direction of the difference when the null hypothesis is rejected. If Dr. Keim had failed to reject the null hypothesis, then there would not have been enough evidence to conclude that a difference exists between the two means, so there would have been no reason to consider the direction of the difference.

APA Format

Reporting the results in APA format for the humidity results, Dr. Keim would write

t (5) = 8.82, p < .05

For a t-test, APA format contains five pieces of information: (1) what test was done, (2) the number of cases, (3) the value of the test statistic, (4) the alpha level chosen, and (5) whether the null hypothesis was rejected.

Cohen’s d and r² for the Paired-Samples t Test

In the last two chapters, for the single-sample t test and the independent-samples t test, we calculated Cohen’s d and r² in order to quantify how much impact the explanatory variable had on the outcome variable. It is possible to calculate Cohen’s d and r² for a paired-samples t test, but doing so is not advisable.

The reason that these effect sizes are inappropriate for a paired-samples t test is that they measure more than just the impact of the explanatory variable on the dependent variable. Cohen’s d and r² for a paired-samples t test also include the effect of individual differences. As a result, they overestimate the size of the effect of the independent variable on the dependent variable. For example, r² would be 94% for the humidity data if it were calculated. If true, it would mean that humidity status explains 94% of the variability in perceived temperature. That would be a huge effect for the explanatory variable of humidity level. Unfortunately, it is not true. Here’s why.

306

Look at Table 9.1, which presents the data for each case in each of the two conditions. Notice that there’s variability within a condition: not everyone perceives the temperature the same way. As everyone in a condition receives the same treatment, the variability within a condition results from individual differences among the participants. Look at case 6 who perceived the low-humidity condition as the coldest, 68°. Case 6 also gives the lowest rating of the temperature in the high-humidity condition. Case 6 differs from the other individuals in this study in that he or she always feels cold. Case 2, in contrast, differs in the perception of temperature, feeling the warmest in both conditions.

The type of participant, cold-sensitive or heat-sensitive, has a large effect on the temperature the environment is perceived to be. Whether a person is cold-sensitive or heat-sensitive is an individual differences variable. When r² or Cohen’s d is calculated for a paired-samples t test, they mix in the impact of these individual differences on temperature with the impact of the different conditions (high vs. low humidity) on temperature. This means that they give an inflated effect size and are not appropriate to estimate effect size for a paired-samples t test.

So, how does a researcher measure effect size for a paired-samples t test? One way is to calculate the confidence interval for the difference between population means and then use professional expertise—and common sense—to translate it into an effect size. Another approach, not possible until Chapter 12, is to use a test called the repeated-measures ANOVA instead of a paired-samples t test. The repeated-measures ANOVA, unlike the paired-samples t test, separates out the impact of individual differences from the effect of the explanatory variable, allowing the effect of the explanatory variable alone to be assessed.

The confidence interval for the difference between population means reveals how small or how large the difference between the population means might be. For the humidity data set, the confidence interval will tell, at a population level, how much hotter the temperature is perceived when the humidity rises. A confidence interval of any percentage could be calculated, but the most common is the 95% confidence interval. Equation 9.5 gives the formula for the 95% confidence interval for the difference between population means for the paired-samples t test.

307

Applying this to her humidity data, Dr. Keim would calculate the 95% confidence interval as follows:

95%CIµ_Diff = (82.50 – 75.00) ± (2.571 × 0.85)

= 7.5000 ± 2.1854

= from 5.3146 to 9.6854

= [5.31, 9.69]

Her confidence interval ranges from a lower limit of 5.31° to an upper limit of 9.69°. To make the interpretation more accessible, Dr. Keim has rounded each end to a whole number, making the confidence interval range from 5 to 10 degrees. In interpreting this confidence interval, Dr. Keim will pay attention to three points: (1) whether the confidence interval captures zero, (2) how close it comes to zero, and (3) how wide it is.

For the humidity study, the confidence interval, from 5 to 10 degrees, does not capture zero. This is to be expected as the null hypothesis was rejected. The confidence interval reiterates what is already known—it is unlikely, in the larger population, that there is no difference in how 76° is perceived in low humidity vs. how it is perceived in high humidity. Instead, in the population, the high-humidity condition is probably perceived as 5–10 degrees hotter, on average, than the same temperature at low humidity.

The second step, determining how small and how large the effect may be, takes more thought and expertise for the paired-samples t test than it did for the independent-samples t test. With an independent-samples t test, the researcher could calculate Cohen’s d or r² and rely on Cohen’s standards for small, medium, and large effects. The end of the confidence interval closer to zero (the lower limit) is 5 and the end farther away (the upper limit) is 10. In the larger population, will people perceive the temperature to be a lot hotter or a little hotter if they perceive 76° in conditions of high humidity as 5 degrees hotter than in low humidity? Dr. Keim believes this to be a meaningful effect. She reasons that, on a summer day, there is a noticeable difference in comfort level between being in an air-conditioned room at 71° and one at 76°, so a 5-degree difference is a meaningful one. If a 5-degree difference is meaningful, then a 10-degree difference is even more so. In understanding what a 10-degree difference means, Dr. Keim thinks of how a 76° summer day feels pleasant and an 86° day feels hot.

308

Finally, she uses Equation 7.5 to calculate the width of the confidence interval. To avoid rounding error, she uses the real limits of the confidence interval, 5.31 to 9.69, not her rounded version of 5 to 10:

CI_W = CI_UL – CI_LL

= 9.69 – 5.31

= 4.38

The confidence interval is 4.38° wide. This seems sufficiently narrow to Dr. Keim. She would like to replicate the study in order to make sure the same effect is observed again, and she would like to increase the sample size in order to have a better sample, but she feels no need to replicate with a larger sample size in order to narrow the confidence interval. Given that both a 5-degree difference and a 10-degree difference seem meaningful and that the confidence interval is narrow, Dr. Keim is inclined to focus on the observed difference of 7.50 degrees in her interpretation.

Here is Dr. Keim’s four-point interpretation in which she states: (1) what the study was about; (2) the main results; (3) what the results mean; and (4) her suggestions for future research. This interpretation is a little longer than previous ones. It takes Dr. Keim two paragraphs to say all that she wants to.

The impact of humidity on perceived temperature was examined. Using a within-subjects design, six participants judged the temperature after being in a 76° room under conditions of low humidity and high humidity. In the low-humidity condition, they judged the temperature fairly accurately (M = 75.00°F), but they judged it as hotter (M = 82.50°F) under conditions of high humidity. This 7.50°F difference was statistically significant [t (5) = 8.82°F, p < .05]. Humidity appears to make people feel hotter. According to this study, people feel almost 8 degrees hotter under conditions of high humidity. This is a noticeable increase in perceived temperature and can move a person from feeling comfortable to being uncomfortable.

There were several limitations to this study that can be rectified in future research. All participants were college students in the United States, people who have experience with heated and cooled environments. It would be interesting to see if the same effect were observed among people with less control over their indoor environments. A second limitation is that only one temperature, 76°, was tested. The effect of humidity on perceived temperature should be observed at both higher and lower temperatures. Finally, there were only six participants in this study. Replicating the study would increase confidence in the robustness of the finding that humidity level affects the perception of temperature.

This is a good opportunity to talk about the difference between statistical significance and practical significance. Statistical significance indicates that the observed difference between sample means is large enough to conclude that there is a difference between population means. It doesn’t mean that the difference is a meaningful one.

Statistical significance is heavily influenced by sample size: the larger the sample size, the more likely it is that results will be statistically significant. If the sample size were increased in the depression follow-up study from 16 to 23, the results would be statistically significant and the confidence interval would range from 0.004 to 2.00, not capturing zero.

Think about the larger population. In the larger population, if the mean depression score were 14.00 at the end of treatment and 14.004 six months later, then the null hypothesis would be wrong and should be rejected. But a population difference of such a small amount, 0.004, would be so small as to be meaningless. It is of no practical significance. A result is of practical significance (also called clinical significance) if the size of the effect is large enough to make a real difference. Practical significance means that the explanatory variable has a meaningful impact on the outcome variable. A statistically significant result is no guarantee of a practically significant effect.

Judging practical significance requires expertise in and familiarity with a specific area of research. Unless one has experience with a scale, it is hard to know how meaningful a 2-point change on the Beck Depression Inventory. For now, the best option is to use Cohen’s small, medium, and large effect sizes as an initial, and very rough, guide to practical and clinical significance.

311