Don Smetzer/Getty Images

EXAMPLE 14.4

Does myth busting really improve public health? Here are some myths and facts.

From the Web site of the Headquarters Counseling Center (2005) in Lawrence,

Kansas:

• Myth: “Suicide happens without warning.”
• Fact: “Most suicidal persons talk about and/or give behavioral clues about their suicidal feelings, thoughts, and intentions.”

From the Web site for the World Health Organization (2007):

• Myth: “Disasters bring out the worst in human behavior.”
• Fact: “Although isolated cases of antisocial behavior exist, the majority of people respond spontaneously and generously.”

A group of Canadian researchers examined the effectiveness of myth busting (Skurnik, Yoon, Park, & Schwarz, 2005). They wondered whether the effectiveness of debunking false medical claims depends on the age of the person targeted by the message. In one study, they compared two groups of adults: younger adults, ages 18–25, and older adults, ages 71–86. Participants were presented with a series of claims and were told that each claim was either true or false. (In reality, all claims were true, partly because researchers did not want to run the risk that participants would misremember false claims as being true.) In some cases, the claim was presented once, and in other cases, it was repeated three times. In either case, the accurate information was presented after each “false” statement. (Note that we have altered the study’s design somewhat to make the study simpler for our purposes.)

The two independent variables in this study were age, with two levels (younger, older), and number of repetitions, with two levels (once, three times). The dependent variable, proportion of responses that were wrong after a 3-day delay, was calculated for each participant. This was a two-way between-groups ANOVA—more specifically, a 2 × 2 between-groups ANOVA. From this name, we know that the table has four cells: (2 × 2) = 4. There were 64 participants—16 in each cell. But here, we use an example with 12 participants—3 in each cell. Here are the data that we’ll use; they have similar means to those in the actual study, and the F statistics are similar as well.

Let’s consider the steps of hypothesis testing for a two-way between-groups ANOVA in the context of this example.

STEP 1: Identify the populations, distribution, and assumptions.

The first step of hypothesis testing for a two-way between-groups ANOVA is very similar to that for a one-way between-groups ANOVA. First, we state the populations, but we specify that they are broken down into more than one category. In the current example, there are four populations, so there are four cells (as shown in Table 14-12). As we do the calculations, the first independent variable, age, appears in the rows of the table, and the second independent variable, number of repetitions, appears in the columns of the table.

There are four populations, each with labels representing the levels of the two independent variables to which they belong.

• Population 1 (Y; 1): Younger adults who hear one repetition of a false claim.
• Population 2 (Y; 3): Younger adults who hear three repetitions of a false claim.
• Population 3 (O; 1): Older adults who hear one repetition of a false claim.
• Population 4 (O; 3): Older adults who hear three repetitions of a false claim.

We next consider the characteristics of the data to determine the distributions to which we compare the sample. We have more than two groups, so we need to consider variances to analyze differences among means. Therefore, we use F distributions.

Finally, we list the hypothesis test that we use for those distributions and check the assumptions for that test. For F distributions, we use ANOVA—in this case, two-way between-groups ANOVA.

The assumptions are the same for all types of ANOVA. The sample should be selected randomly; the populations should be distributed normally; and the population variances should be equal. Let’s explore that a bit further.

(1) These data were not randomly selected. Younger adults were recruited from a university, and older adults were recruited from the local community. Because random sampling was not used, we must be cautious when generalizing from these samples. (2) The researchers did not report whether they investigated the shapes of the distributions of their samples to assess the shapes of the underlying populations. (3) The researchers did not provide standard deviations of the samples as an indication of whether the population spreads might be approximately equal—the condition known as homoscedasticity, which we explored in chapter 12. We typically explore these assumptions using the sample data.

Summary: Population 1 (Y; 1): Younger adults who hear one repetition of a false claim. Population 2 (Y; 3): Younger adults who hear three repetitions of a false claim. Population 3 (O; 1): Older adults who hear one repetition of a false claim. Population 4 (O; 3): Older adults who hear three repetitions of a false claim.

The comparison distributions will be F distributions. The hypothesis test will be a two-way between-groups ANOVA. Assumptions: (1) The data are not from random samples, so we must generalize only with caution. (2) From the published research report, we do not know if the underlying population distributions are normal. (3) We do not know if the population variances are approximately equal (homoscedasticity).

STEP 2: State the null and research hypotheses.

The second step, to state the null and research hypotheses, is similar to that for a one-way between-groups ANOVA, except that we now have three sets of hypotheses, one for each main effect and one for the interaction. Those for the two main effects are the same as those for the one effect of a one-way between-groups ANOVA (see the summary below). If there are only two levels, then we can simply say that the two levels are not equal; if there are only two levels and there is a statistically significant difference, the difference must be between those two levels. Note that because there are two independent variables, we clarify which variable we are referring to by using initial letters or abbreviations for the levels of each (e.g., Y for younger and O for older). If an independent variable has more than two levels, the research hypothesis would be that any two levels of the independent variable are not equal.

The hypotheses for the interaction are typically stated in words but not in symbols. The null hypothesis is that the effect of one independent variable is not dependent on the levels of the other independent variable. The research hypothesis is that the effect of one independent variable depends on the levels of the other independent variable. It does not matter which independent variable we list first (e.g., “the effect of age is not dependent…” or “the effect of number of repetitions is not dependent…”). Write the hypotheses in the way that makes the most sense to you.

Summary: The hypotheses for the main effect of the first independent variable, age, are as follows. Null hypothesis: On average, compared with older adults, younger adults have the same proportion of responses that are wrong when remembering which claims are myths—H₀: μ_Y = μ_O. Research hypothesis: On average, compared with older adults, younger adults have a different proportion of responses that are wrong when remembering which claims are myths—H₁: μ_Y ≠ μ_O.

The hypotheses for the main effect of the second independent variable, number of repetitions, are as follows. Null hypothesis: On average, those who hear one repetition have the same proportion of responses that are wrong when remembering which claims are myths compared with those who hear three repetitions—H₀: μ₁ = μ₃. Research hypothesis: On average, those who hear one repetition have a different proportion of responses that are wrong when remembering which claims are myths compared with those who hear three repetitions—H₁: μ₁ ≠ μ₃.

The hypotheses for the interaction of age and number of repetitions are as follows. Null hypothesis: The effect of number of repetitions is not dependent on the levels of age. Research hypothesis: The effect of number of repetitions depends on the levels of age.

STEP 3: Determine the characteristics of the comparison distribution.

The third step is similar to that of a one-way between-groups ANOVA, except that there are three comparison distributions, all of them F distributions. We need to provide the appropriate degrees of freedom for each of these: two main effects and one interaction. As before, each F statistic is a ratio of between-groups variance and within-groups variance. Because there are three effects, there are three between-groups variance estimates, each with its own degrees of freedom. There is only one within-groups variance estimate, with its degrees of freedom for all three.

For each main effect, the between-groups degrees of freedom is calculated as for a one-way ANOVA: the number of groups minus 1. The first independent variable, age, is in the rows of the table of cells, so the between-groups degrees of freedom is:

The second independent variable, number of repetitions, is in the columns of the table of cells, so the between-groups degrees of freedom is:

We now need a between-groups degrees of freedom for the interaction, which we calculate by multiplying the degrees of freedom for the two main effects:

The within-groups degrees of freedom is calculated like that for a one-way between-groups ANOVA, the sum of the degrees of freedom in each of the cells. In the current example, there are three participants in each cell, so the within-groups degrees of freedom is calculated as follows, with N representing the number in each cell:

For a check on our work, we calculate the total degrees of freedom just as we did for the one-way between-groups ANOVA. We subtract 1 from the total number of participants:

We now add up the three between-groups degrees of freedom and the within-groups degrees of freedom to see if they equal 11. In this case, they match:

Finally, for this step, we list the distributions with their degrees of freedom for the three effects. Note that, although the between-groups degrees of freedom for the three effects are the same in this case, they are often different. For example, if one independent variable had three levels and the other had four, the between-groups degrees of freedom for the main effects would be 2 and 3, respectively, and the between-groups degrees of freedom for the interaction would be 6.

Summary: Main effect of age: F distribution with 1 and 8 degrees of freedom. Main effect of number of repetitions: F distribution with 1 and 8 degrees of freedom. Interaction of age and number of repetitions: F distribution with 1 and 8 degrees of freedom. (Note: It is helpful to include all degrees of freedom calculations in this step.)

STEP 4: Determine the critical values, or cutoffs.

Again, this step for the two-way between-groups ANOVA is just an expansion of that for the one-way version. We now need three critical values but they’re determined just as we determined them before. We use the F table in Appendix B.

For each main effect and for the interaction, we look up the within-groups degrees of freedom, which is always the same for each effect, along the left-hand side and the appropriate between-groups degrees of freedom across the top of the table. The place on the grid where this row and this column intersect contains three numbers. From top to bottom, the table provides cutoffs for p levels of 0.01, 0.05, and 0.10. As usual, we typically use 0.05. In this instance, it happens that the critical value is the same for all three effects because the between-groups degrees of freedom is the same for all three. But when the between-groups degrees of freedom are different, there are different critical values. Here, we look up the between-groups degrees of freedom of 1, within-groups degrees of freedom of 8, and p level of 0.05. The cutoff for all three is 5.32, as seen in Figure 14-8.

Summary: There are three critical values (which in this case are all the same), as seen in the curve in Figure 14-8. The critical F value for the main effect of age is 5.32. The critical F value for the main effect of number of repetitions is 5.32. The critical F value for the interaction of age and number of repetitions is 5.32.

STEP 5: Calculate the test statistic.

As with the one-way between-groups ANOVA, the fifth step for the two-way between-groups ANOVA is the most time consuming. As you might guess, it’s similar to what we already learned, but we have to calculate three F statistics instead of one. We learn the logic and the specific calculations for this step in the next section.

STEP 6: Make a decision.

This step is the same as for a one-way between-groups ANOVA, except that we compare each of the three F statistics to its appropriate cutoff F statistic. If the F statistic is beyond the critical value, then we know that it is in the most extreme 5% of possible test statistics if the null hypothesis is true. After making a decision for each F statistic, we present the results in one of three ways.

First, if we are able to reject the null hypothesis for the interaction, then we draw a specific conclusion with the help of a table and graph. Because we have more than two groups, we use a post hoc test, such as the one that we learned in Chapter 12. When there are three effects, post hoc tests are typically implemented separately for each main effect and for the interaction (Hays, 1994). If the interaction is statistically significant, then it might not matter whether the main effects are also significant; if they are also significant, then those findings are usually qualified by the interaction, and are not described separately. The overall pattern of cell means tells the whole story.

Second, if we are not able to reject the null hypothesis for the interaction, then we focus on any significant main effects, drawing a specific directional conclusion for each. In this study, each independent variable has only two levels, so there is no need for a post hoc test. If there were three or more levels, however, then each significant main effect would require a post hoc test to determine exactly where the differences lie. Third, if we do not reject the null hypothesis for either main effect or the interaction, then we can only conclude that there is insufficient evidence from this study to support the research hypotheses. We will complete step 6 of hypothesis testing for this study in the next section, after we consider the calculations of the source table for a two-way between-groups ANOVA.

MASTERING THE FORMULA

14-6: We calculate the total sum of squares using the following formula: SS_total = Σ(X − GM)². We subtract the grand mean from every score to create deviations, then square the deviations, and finally sum the squared deviations.

Table : TABLE 14-13. Calculating the Total Sum of Squares The total sum of squares is calculated by subtracting the overall mean, called the grand mean, from every score to create deviations, then squaring the deviations and summing them: Σ(X − GM)² = 0.0672

MASTERING THE FORMULA

14-7: We calculate the between-groups sum of squares for the first independent variable, that in the rows of the table of cells, using the following formula: SS_{between(rows)} = Σ(M_row − GM)². For every participant, we subtract the grand mean from the marginal mean for the appropriate row for that participant. We square these deviations, and sum the squared deviations.

Table : TABLE 14-14. Means for False Medical Claims Study The study of the misremembering of false medical claims as true had two independent variables, age and number of repetitions. The cell means and marginal means for error rates are shown in the table. The grand mean is 0.21.

Table : TABLE 14-15. Calculating the Sum of Squares for the First Independent Variable The sum of squares for the first independent variable is calculated by subtracting the overall mean (the grand mean) from the mean for each level of that variable—in this case, age—to create deviations, then squaring the deviations and summing them: Σ(M_row(age) − GM)² = 0.03.

MASTERING THE FORMULA

14-8: We calculate the between-groups sum of squares for the second independent variable, that in the columns of the table of cells, using the following formula: SS_{between(columns)} = Σ(M_column(reps) − GM)². For every participant, we subtract the grand mean from the marginal mean for the appropriate column for that participant. We square these deviations, and then sum the squared deviations.

Table : TABLE 14-16. Calculating the Sum of Squares for the Second Independent Variable The sum of squares for the second independent variable is calculated by subtracting the overall mean (the grand mean) from the mean for each level of that variable—in this case, number of repetitions—to create deviations, then squaring the deviations and summing them: Σ(M_column(reps) − GM)² = 0.

MASTERING THE FORMULA

14-9: We calculate the within-groups sum of squares using the following formula: SS_within = Σ(X − M_cell)². For every participant, we subtract the appropriate cell mean from that participant’s score. We square these deviations, and sum the squared deviations.

Table : TABLE 14-17. Calculating the Within-Groups Sum of Squares The within-groups sum of squares is calculated the same way for a two-way ANOVA as for a one-way ANOVA. We take each score and subtract the mean of the cell from which it comes—not the grand mean—to create deviations; then we square the deviations and sum them: Σ(X − M_cell)² = 0.018.

MASTERING THE FORMULA

14-10: To calculate the between-groups sum of squares for the interaction, we subtract the two between-groups sums of squares for the independent variables and the within-groups sum of squares from the total sum of squares. The formula is: SS_{between(interaction)} = SS_total − (SS_{between(rows)} + SS_{between(columns)} + SS_within).

MASTERING THE FORMULA

14-11: The formulas to calculate the four mean squares are in Table 14-18. There are three between-groups mean squares—one for each main effect and one for the interaction—and one within-groups mean square. For each mean square, we divide the appropriate sum of squares by its related degrees of freedom. The formulas for the three F statistics, one for each main effect and one for the interaction, are also in Table 14-18. For each of the three effects, we divide the appropriate between-groups mean square by the within-group mean square. The denominator is the same in all three cases.

Table : TABLE 14-19. The Expanded Source Table and False Medical Claims This expanded source table shows the actual sums of squares, degrees of freedom, mean squares, and F statistics for the study on false medical claims.

MASTERING THE FORMULA

14-12: To calculate effect size for two-way ANOVA, we make three R² calculations, one for each main effect and one for the interaction. In each case, we divide the ap propriate between-groups sum of squares by the total sum of squares minus the sums of squares for the other two effects. For example, the effect size for the interaction is calculated using this formula: .

EXAMPLE 14.5

Let’s apply this to the ANOVA we just conducted. We use the statistics in the source table shown in Table 14-19 to calculate R² for each main effect and the interaction. Here are the calculations for the main effect for age:

Here are the calculations for the main effect for repetitions:

Here are the calculations for the interaction:

The conventions are the same as those presented in Chapter 12, shown again here in Table 14-20. From this table, we can see that the R² of 0.63 for the main effect of age and 0.52 for the interaction are very large. The R² of 0.00 for the main effect of repetitions indicates that there is no observable effect in this study.

Next Steps

Variations on ANOVA

We’ve already seen the flexibility that ANOVA offers in terms of both independent variables and research design. Yet ANOVA is even more flexible than we’ve seen so far in this and the previous two chapters. We described within-groups ANOVAs in which the participants experience all of the research conditions. Researchers also use four slightly more complicated designs.

1. A mixed-design ANOVA is used to analyze the data from a study with at least two independent variables; at least one variable must be within-groups and at least one variable must be between-groups. In other words, a mixed design includes both a between-groups variable and a within-groups variable.
2. A multivariate analysis of variance (MANOVA) is a form of ANOVA in which there is more than one dependent variable. The word multivariate refers to the number of dependent variables, not the number of independent variables. (Remember, a plain old ANOVA already can handle multiple independent variables.)
3. Analysis of covariance (ANCOVA) is a type of ANOVA in which a covariate is included so that statistical findings reflect effects after a scale variable has been statistically removed. Specifically, a covariate is a scale variable that we suspect associates, or covaries, with the independent variable of interest. So ANCOVA statistically subtracts the effect of a possible confounding variable.
4. We can also combine the features of a MANOVA and an ANCOVA. A multivariate analysis of covariance (MANCOVA) is an ANOVA with multiple dependent variables and a covariate. MANOVAs, ANCOVAs, and MANCOVAs can each have a between-groups design, a within-groups design, or even a mixed design. Table 14-21 shows variations on ANOVA.

Let’s consider an example of a mixed-design ANOVA. In Chapter 12, we discussed a study about the effects of different types of e-mails on final exam grades for two groups of students, those with a grade of C on the first exam and those with a grade of D or F on the first exam (Forsyth, Lawrence, Burnette, & Baumeister, 2007). When the researchers first presented these data, they conducted a three-way ANOVA; they used three independent variables. The first was the same as those described in the example in Chapter 12: type of e-mail [the control group (no message), the self-esteem group, and the take-responsibility group]. The second was initial grade (C and D/F). Both of these independent variables are between-groups.

However, these researchers also included a third independent variable in their analyses. They included not only final exam grades but also grades from the earlier midterm exam. So they had another independent variable, exam, with two levels: midterm and final. Because every student took both exams, this third independent variable is within-groups. This means that the research design had two between-groups independent variables and one within-groups independent variable. This is an example of a mixed-design ANOVA. Specifically, this ANOVA would be referred to as a 2 (grade: C, D/F) × 3 (type of e-mail: control, self-esteem, take responsibility) × 2 (exam: midterm, final) mixed-design ANOVA.

Let’s consider an example of a MANCOVA, an analysis that includes both (a) multiple dependent variables and (b) at least one covariate.

(a) We sometimes use a multivariate analysis when we have several similar dependent variables. Aside from the use of multiple dependent variables, multivariate analyses are not all that different from those with one dependent variable. Essentially, the calculations treat the group of dependent variables as one dependent variable. Although we can follow up a MANOVA by considering the different univariate (single dependent variable) ANOVAs embedded in the MANOVA, we often are most interested in the effect of the independent variables on the composite of dependent variables.

(b) There are often situations in which we suspect that a third variable might be affecting the dependent variable. In these cases, we might conduct an ANCOVA or MANCOVA. We might, for example, have level of education as one of the independent variables and worry that age, which is likely related to level of education, is actually what is influencing the dependent variable, not education. In this case, we could include age as a covariate.

The inclusion of a covariate means that the analysis will look at the effects of the independent variables on the dependent variables after statistically removing the effect of one or more third variables. At its most basic, conducting an ANCOVA is almost like conducting an ANOVA at each level of the covariate. If age were the covariate with level of education as the independent variable and income as the dependent variable, then we’d essentially be looking at a regular ANOVA for each age. We want to answer the question: Given a certain age, does education predict income? Of course, this is a simplified explanation, but that’s the logic behind the procedure. If the calculations find that education has an effect on income among 33-year-olds, 58-year-olds, and every other age group, then we know that there is a main effect of education on income, over and above the effect of age.

Researchers conducted a MANCOVA to analyze the results of a study examining military service and marital status within the context of men’s satisfaction within their romantic relationships (McLeland & Sutton, 2005). The independent variables were military service (military, nonmilitary) and marital status (married, unmarried). There were two dependent variables, both measures of relationship satisfaction: the Kansas Marital Satisfaction Scale (KMSS) and the ENRICH Marital Satisfaction Scale (EMS). The researchers also included the covariate of age.

Initial analyses also found that age was significantly associated with relationship satisfaction: Older men tended to be more satisfied than younger men. The researchers wanted to be certain that it was military status and marital status, not age, that affected relationship satisfaction, so they controlled for age as a covariate. The MANCOVA led to only one statistically significant finding: Military men were less satisfied than nonmilitary men with respect to their relationships, when controlling for the age of the men. That is, given a certain age, military men of that age are likely to be less satisfied with their relationships than are nonmilitary men of that age.

CHECK YOUR LEARNING

Clarifying the Concepts

• 14-9 What is the basic difference between the six steps of hypothesis testing for a two-way between-groups ANOVA and a one-way between-groups ANOVA?
• 14-10 What are the four sources of variability in a two-way ANOVA?

Calculating the Statistics

• 14-11 Compute the three between-groups degrees of freedom (both main effects and the interaction), the within-groups degrees of freedom, and the total degrees of freedom for the following data:
  IV 1, level A; IV 2, level A: 2, 1, 1, 3
  IV 1, level B; IV 2, level A: 5, 4, 3, 4
  IV 1, level A; IV 2, level B: 2, 3, 3, 3
  IV 1, level B; IV 2, level B: 3, 2, 2, 3
  
• 14-12 Using the degrees of freedom you calculated in Check Your Learning 14-11, determine critical values, or cutoffs, using a p level of 0.05, for the F statistics of the two main effects and the interaction.

Applying the Concepts

• 14-13 Researchers studied the effect of e-mail messages on students’ final exam grades (Forsyth & Kerr, 1999; Forsyth et al., 2007). To test for possible interactions, participants included students whose first exam grade was either (1) a C, or (2) a D or an F. Participants were randomly assigned to receive several e-mails in one of three conditions: e-mails intended to bolster their self-esteem, e-mails intended to enhance their sense of control over their grades, and e-mails that just included review questions (control group). The accompanying table shows the cell means for the final exam grades (note that some of these are approximate, but all represent actual findings). For simplicity, assume there were 84 participants in the study evenly divided among cells.

  	Self-Esteem (SE)	Take Responsibility (TR)	Control Group (CG)
  C	67.31	69.83	71.12
  D/F	47.83	60.98	62.13

  1. From step 1 of hypothesis testing, list the populations for this study.
  2. Conduct step 2 of hypothesis testing.
  3. Conduct step 3 of hypothesis testing.
  4. Conduct step 4 of hypothesis testing.
  5. The F statistics are 20.84 for the main effect of the independent variable of initial grade, 1.69 for the main effect of the independent variable of type of e-mail, and 3.02 for the interaction. Conduct step 6 of hypothesis testing.