The ANOVA test gives a general answer to a general question: are the differences among observed group means significant? Unfortunately, a small -value simply tells us that the group means are not all the same. It does not tell us specifically which means differ from each other. Plotting and inspecting the means give us some indication of where the differences lie, but we would like to supplement inspection with formal inference. This section presents two approaches to the task of comparing group means.

Contrasts

The preferred approach is to pose specific questions regarding comparisons among the means before the data are collected. We can answer specific questions of this kind and attach a level of confidence to the answers we give. We now explore these ideas in the setting of Case 14.2.

We can compare the new with the standard by posing and answering specific questions about the mean responses. First, here is the basic ANOVA.

The ANOVA null hypothesis is

where the subscripts correspond to the group labels Basal, DRTA, and Strat. Figure 14.14 shows that with degrees of freedom 2 and 63. The -value is 0.015. We have strong evidence against .

736

What can the researchers conclude from this analysis? The alternative hypothesis is true if or if or if or if any combination of these statements is true. We would like to be more specific.

Each of and says that a combination of population means is 0. These combinations of means are called contrasts. We use the Greek letter psi, for contrasts among population means. The two contrasts that arise from our two null hypotheses are

and

In each case, the value of the contrast is 0 when is true. We chose to define the contrasts so that they will be positive when the alternative hypothesis is true. Whenever possible, this is a good idea because it makes some computations easier.

A contrast expresses an effect in the population as a combination of population means. To estimate the contrast, form the corresponding sample contrast by using sample means in place of population means. Under the ANOVA assumptions, a sample contrast is a linear combination of independent Normal variables and, therefore, has a Normal distribution. We can obtain the standard error of a contrast by using the rules for variances. Inference is based on statistics. Here are the details.

737

Because each estimates the corresponding , the addition rule for means tells us that the mean of the sample contrast is . In other words, is an unbiased estimator of . Testing the hypothesis that a contrast is 0 assesses the significance of the effect measured by the contrast. It is often more informative to estimate the size of the effect using a confidence interval for the population contrast.

We look at inference for each of these contrasts in turn.

The statistic for testing versus is

Because has 63 degrees of freedom, software using the distribution gives the one-sided -value as 0.0044. If we used Table D, we would conclude that . The -value is small, so there is strong evidence against . The researchers have shown that the new methods produce higher mean scores than the old. The size of the improvement can be described by a confidence interval. To find the 95% confidence interval for , we combine the estimate with its margin of error:

The interval is (1.15, 7.75). We are 95% confident that the mean improvement obtained by using one of the innovative methods rather than the old method is between 1.15 and 7.75 points.

The statistic for assessing the significance of this contrast is

The -value for the two-sided alternative is 0.2020. We conclude that either the two new methods have the same population means or the sample sizes are not sufficiently large to distinguish them. A confidence interval helps clarify this statement. To find the 95% confidence interval for , we combine the estimate with its margin of error:

The interval is (−1.34, 6.26). With 95% confidence we state that the difference between the population means for the two new methods is between −1.34 and 6.26.

739

Some statistical software packages report the test statistics associated with contrasts as statistics rather than statistics. These statistics are the squares of the statistics described earlier. The associated -values are for the two-sided alternatives.

Questions about population means are expressed as hypotheses about contrasts. A contrast should express a specific question that we have in mind when designing the study. When contrasts are formulated before seeing the data, inference about contrasts is valid whether or not the ANOVA of equality of means is rejected. Because the test answers a very general question, it is less powerful than tests for contrasts designed to answer specific questions. Specifying the important questions before the analysis is undertaken enables us to use this powerful statistical technique.

740

Multiple comparisons

In many studies, specific questions cannot be formulated in advance of the analysis. If is not rejected, we conclude that the population means are indistinguishable on the basis of the data given. On the other hand, if is rejected, we would like to know which pairs of means differ. Multiple-comparisons methods address this issue. It is important to keep in mind that multiple-comparisons methods are commonly used only after rejecting the ANOVA .

Return once more to the reading comprehension study described in Case 14.2. We found in Example 14.15 (pages 734–735) that the means were not all the same ( and 63, ).

These statistics are very similar to the pooled two-sample statistic for comparing two population means. The difference is that we now have more than two populations, so each statistic uses the pooled estimator from all groups rather than the pooled estimator from just the two groups being compared. This additional information about the common increases the power of the tests. The degrees of freedom for all of these statistics are , those associated with .

Because we do not have any specific ordering of the means in mind as an alternative to equality, we must use a two-sided approach to the problem of deciding which pairs of means are significantly different.

One obvious choice for is the upper critical value for the distribution. This choice simply carries out as many separate significance tests of fixed level as there are pairs of means to be compared. The procedure based on this choice is called the least-significant differences method, or simply LSD. LSD has some undesirable properties, particularly if the number of means being compared is large. Suppose, for example, that there are groups and we use LSD with . There are 190 different pairs of means. If we perform 190 tests, each with an error rate of 5%, our overall error rate will be unacceptably large. We would expect about 5% of the 190 to be significant even if the corresponding population means are all the same. Because 5% of 190 is 9.5, we would expect 9 or 10 false rejections.

The LSD procedure fixes the probability of a false rejection for each single pair of means being compared. It does not control the overall probability of some false rejection among all pairs. Other choices of control possible errors in other ways. The choice of is, therefore, a complex problem, and a detailed discussion of it is beyond the scope of this text. Many choices for are used in practice. One major statistical package allows selection from a list of more than a dozen choices.

We discuss only one of these, the Bonferroni method. Use of this procedure with , for example, guarantees that the probability of any false rejection among all comparisons made is no greater than 0.05. This is much stronger protection than controlling the probability of a false rejection at 0.05 for each separate comparison.

Usually, we use software to perform the multiple-comparisons procedure. The formats differ from package to package but they all give the same basic information.

742

SPSS does not give the values of the statistics for multiple comparisons. To compute them, simply divide the difference in the means by the standard error. For comparing Basal with DRTA, we have (as before)

When there are many groups, the many results of multiple comparisons are difficult to describe. Here is one common format.

The display in Example 14.25 shows that, at the overall 0.05 significance level, Basal does not differ from Strat and Strat does not differ from DRTA, yet Basal does differ from DRTA. These conclusions appear to be illogical. If is the same as , and is the same as , doesn’t it follow that is the same as ? Logically, the answer must be Yes.

This apparent contradiction points out the nature of the conclusions of tests of significance. A careful statement would say that we found significant evidence that Basal differs from DRTA and failed to find evidence that Basal differs from Strat or that Strat differs from DRTA. Failing to find strong enough evidence that two means differ doesn’t say that they are equal. It is very unlikely that any two methods of teaching reading comprehension would give exactly the same population means, but the data can fail to provide good evidence of a difference. This is particularly true in multiple-comparisons methods such as Bonferroni that use a single for an entire set of comparisons.

744

Simultaneous confidence intervals

One way to deal with these difficulties of interpretation is to give confidence intervals for the differences. The intervals remind us that the differences are not known exactly. We want to give simultaneous confidence intervals—that is, intervals for all the differences among the population means with, say, 95% confidence that all the intervals at once cover the true population differences. Again, there are many competing procedures—in this case, many methods of obtaining simultaneous intervals.

The confidence intervals generated by a particular choice of are closely related to the multiple comparisons results for that same method. If one of the confidence intervals includes the value 0, then that pair of means will not be declared significantly different, and vice versa.