Does myth busting really improve public health? Here are some myths and facts.

From the Web site of the Canadian Mental Health Association (2015):

Myth: “People who experience mental illnesses can’t work.”

Fact: “Whether you realize it or not, workplaces are filled with people who have experienced mental illnesses. Mental illnesses don’t mean that someone is no longer capable of working.”

From the Web site for the World Health Organization (2015):

Myth: “Disasters bring out the worst in human behavior.”

Fact: “Although isolated cases of antisocial behavior exist, the majority of people respond spontaneously and generously.”

A group of Canadian researchers examined the effectiveness of myth busting (Skurnik, Yoon, Park, & Schwarz, 2005). They wondered whether the effectiveness of debunking false medical claims depends on the age of the person targeted by the message. In one study, they compared two groups of adults: younger adults, ages 18–25, and older adults, ages 71–86. Participants were presented with a series of claims and were told that each claim was either true or false. (In reality, all claims were true, partly because researchers did not want to run the risk that participants would misremember false claims as being true.) In some cases, the claim was presented once, and in other cases, it was repeated three times. In either case, the accurate information was presented after each “false” statement. (Note that for our teaching purposes, we have slightly altered the study’s design.)

The two independent variables in this study were age, with two levels (younger, older), and number of repetitions, with two levels (once, three times). The dependent variable, proportion of responses that were wrong after a 3-day delay, was calculated for each participant. This was a two-way between-groups ANOVA—more specifically, a 2 × 2 between-groups ANOVA. From this name, we know that the table has four cells: (2 × 2) = 4. There were 64 participants—16 in each cell. But here we use an example with 12 participants—3 in each cell. Here are the data that we’ll use; they have similar means to those in the actual study, and the F statistics are similar as well.

Let’s consider the steps of hypothesis testing for a two-way between-groups ANOVA in the context of this example.

STEP 1: Identify the populations, distribution, and assumptions.

The first step of hypothesis testing for a two-way between-groups ANOVA is very similar to that for a one-way between-groups ANOVA. First, we state the populations, but we specify that they are broken down into more than one category. In the current example, there are four populations, so there are four cells (as shown in Table 12-12). As we do the calculations, the first independent variable, age, appears in the rows of the table, and the second independent variable, number of repetitions, appears in the columns of the table.

342

There are four populations, each with labels representing the levels of the two independent variables to which they belong.

Population 1 (Y; 1): Younger adults who hear one repetition of a false claim

Population 2 (Y; 3): Younger adults who hear three repetitions of a false claim

Population 3 (O; 1): Older adults who hear one repetition of a false claim

Population 4 (O; 3): Older adults who hear three repetitions of a false claim

We next consider the characteristics of the data to determine the distributions to which we compare the sample. We have more than two groups, so we need to consider variances to analyze differences among means. Therefore, we use F distributions. Finally, we list the hypothesis test that we use for those distributions and check the assumptions for that test. For F distributions, we use ANOVA—in this case, two-way between-groups ANOVA.

The assumptions are the same for all types of ANOVA: The sample should be selected randomly; the populations should be distributed normally; and the population variances should be equal. Let’s explore that a bit further.

(1) These data were not randomly selected. Younger adults were recruited from a university, and older adults were recruited from the local community. Because random sampling was not used, we must be cautious when generalizing from these samples. (2) The researchers did not report whether they investigated the shapes of the distributions of their samples to assess the shapes of the underlying populations. (3) The researchers did not provide standard deviations of the samples as an indication of whether the population spreads might be approximately equal—the condition known as homoscedasticity, which we explored in Chapter 12. We typically explore these assumptions using the sample data.

Summary: Population 1 (Y; 1): Younger adults who hear one repetition of a false claim. Population 2 (Y; 3): Younger adults who hear three repetitions of a false claim. Population 3 (O; 1): Older adults who hear one repetition of a false claim. Population 4 (O; 3): Older adults who hear three repetitions of a false claim.

The comparison distributions will be F distributions. The hypothesis test will be a two-way between-groups ANOVA. Assumptions: (1) The data are not from random samples, so we must generalize only with caution. (2) From the published research report, we do not know if the underlying population distributions are normal. (3) We do not know if the population variances are approximately equal (homoscedasticity).

STEP 2: State the null and research hypotheses.

The second step, to state the null and research hypotheses, is similar to that for a one-way between-groups ANOVA, except that we now have three sets of hypotheses, one for each main effect and one for the interaction. Those for the two main effects are the same as those for the one effect of a one-way between-groups ANOVA (see the summary below). If there are only two levels, then we can simply say that the two levels are not equal; if there are only two levels and there is a statistically significant difference, the difference must be between those two levels. Note that because there are two independent variables, we clarify which variable we are referring to by using initial letters or abbreviations for the levels of each (e.g., Y for younger and O for older). If an independent variable has more than two levels, the research hypothesis would be that any two levels of the independent variable are not equal.

343

The hypotheses for the interaction are typically stated in words but not in symbols. The null hypothesis is that the effect of one independent variable is not dependent on the levels of the other independent variable. The research hypothesis is that the effect of one independent variable depends on the levels of the other independent variable. It does not matter which independent variable we list first (e.g., “The effect of age is not dependent. . .” or “The effect of number of repetitions is not dependent. . .”). Write the hypotheses in the way that makes the most sense to you.

Summary: The hypotheses for the main effect of the first independent variable, age, are as follows. Null hypothesis: On average, compared with older adults, younger adults have the same proportion of responses that are wrong when remembering which claims are myths—H₀: µ_Y = µ_O. Research hypothesis: On average, compared with older adults, younger adults have a different proportion of responses that are wrong when remembering which claims are myths—H₁: µ_Y ≠ µ_O.

The hypotheses for the main effect of the second independent variable, number of repetitions, are as follows. Null hypothesis: On average, those who hear one repetition have the same proportion of responses that are wrong when remembering which claims are myths compared with those who hear three repetitions—H₀: µ₁ = µ₃. Research hypothesis: On average, those who hear one repetition have a different proportion of responses that are wrong when remembering which claims are myths compared with those who hear three repetitions—H₁: µ₁ ≠ µ₃.

The hypotheses for the interaction of age and number of repetitions are as follows. Null hypothesis: The effect of number of repetitions is not dependent on the levels of age. Research hypothesis: The effect of number of repetitions depends on the levels of age.

STEP 3: Determine the characteristics of the comparison distribution.

The third step is similar to that of a one-way between-groups ANOVA, except that there are three comparison distributions, all of them F distributions. We need to provide the appropriate degrees of freedom for each of these: two main effects and one interaction. As before, each F statistic is a ratio of between-groups variance and within-groups variance. Because there are three effects, there are three between-groups variance estimates, each with its own degrees of freedom. There is only one within-groups variance estimate, with its degrees of freedom for all three.

For each main effect, the between-groups degrees of freedom is calculated as for a one-way ANOVA: the number of groups minus 1. The first independent variable, age, is in the rows of the table of cells, so the between-groups degrees of freedom is:

df_rows(age) = N_rows − 1 = 2 − 1 = 1

The second independent variable, number of repetitions, is in the columns of the table of cells, so the between-groups degrees of freedom is:

df_{columns(reps)} = N_columns − 1 = 2 − 1 = 1

344

We now need a between-groups degrees of freedom for the interaction, which we calculate by multiplying the degrees of freedom for the two main effects:

df_interaction = (df_rows(age))(df_{columns(reps)}) = (1)(1) = 1

The within-groups degrees of freedom is calculated like that for a one-way between-groups ANOVA, the sum of the degrees of freedom in each of the cells. In the current example, there are three participants in each cell, so the within-groups degrees of freedom is calculated as follows, with N representing the number in each cell:

df_Y,1 = N − 1 = 3 − 1 = 2

df_Y,3 = N − 1 = 3 − 1 = 2

df_O,1 = N − 1 = 3 − 1 = 2

df_O,3 = N − 1 = 3 − 1 = 2

df_within = df_Y,1 + df_Y,3 + df_O,1 + df_O,3 = 2 + 2 + 2 + 2 = 8

For a check on our work, we calculate the total degrees of freedom just as we did for the one-way between-groups ANOVA. We subtract 1 from the total number of participants:

df_total = N_total − 1 = 12 − 1 = 11

We now add up the three between-groups degrees of freedom and the within-groups degrees of freedom to see if they equal 11. In this case, they match:

11 = 1 + 1 + 1 +8

Finally, for this step, we list the distributions with their degrees of freedom for the three effects. Note that, although the between-groups degrees of freedom for the three effects are the same in this case, they are often different. For example, if one independent variable had three levels and the other had four, the between-groups degrees of freedom for the main effects would be 2 and 3, respectively, and the between-groups degrees of freedom for the interaction would be 6.

Summary: Main effect of age: F distribution with 1 and 8 degrees of freedom. Main effect of number of repetitions: F distribution with 1 and 8 degrees of freedom. Interaction of age and number of repetitions: F distribution with 1 and 8 degrees of freedom. (Note: It is helpful to include all degrees of freedom calculations in this step.)

STEP 4: Determine the critical values, or cutoffs.

Again, this step for the two-way between-groups ANOVA is just an expansion of that for the one-way version. We now need three critical values but we determine them just as we determined them before. We use the F table in Appendix B.

For each main effect and for the interaction, we look up the within-groups degrees of freedom, which is always the same for each effect, along the left-hand side and the appropriate between-groups degrees of freedom across the top of the table. The place on the grid where this row and this column intersect contains three numbers. From top to bottom, the table provides cutoffs for p levels of 0.01, 0.05, and 0.10. As usual, we typically use 0.05. In this instance, it happens that the critical value is the same for all three effects because the between-groups degrees of freedom is the same for all three. But when the between-groups degrees of freedom are different, there are different critical values. Here, we look up the between-groups degrees of freedom of 1, within-groups degrees of freedom of 8, and p level of 0.05. The cutoff for all three is 5.32, as seen in Figure 12-8.

345

Summary: There are three critical values (which in this case are all the same), as seen in the curve in Figure 12-8. The critical F value for the main effect of age is 5.32. The critical F value for the main effect of number of repetitions is 5.32. The critical F value for the interaction of age and number of repetitions is 5.32.

STEP 5: Calculate the test statistic.

As with the one-way between-groups ANOVA, the fifth step for the two-way between-groups ANOVA is the most time consuming. As you might guess, it’s similar to what we already learned, but we have to calculate three F statistics instead of one. We learn the logic and the specific calculations for this step in the next section.

STEP 6: Make a decision.

This step is the same as for a one-way between-groups ANOVA, except that we compare each of the three F statistics to its appropriate cutoff F statistic. If the F statistic is beyond the critical value, then we know that it is in the most extreme 5% of possible test statistics if the null hypothesis is true. After making a decision for each F statistic, we present the results in one of three ways.

First, if we are able to reject the null hypothesis for the interaction, then we draw a specific conclusion with the help of a table and graph. Because we have more than two groups, we use a post hoc test, such as the one that we learned in Chapter 11. When there are three effects, post hoc tests are typically implemented separately for each main effect and for the interaction (Hays, 1994). If the interaction is statistically significant, then it might not matter whether the main effects are also significant; if they are also significant, then those findings are usually qualified by the interaction, and are not described separately. The overall pattern of cell means tells the whole story.

Second, if we are not able to reject the null hypothesis for the interaction, then we focus on any significant main effects, drawing a specific directional conclusion for each. In this study, each independent variable has only two levels, so there is no need for a post hoc test. If there were three or more levels, however, then each significant main effect would require a post hoc test to determine exactly where the differences lie.

Third, if we do not reject the null hypothesis for either main effect or the interaction, then we can only conclude that there is insufficient evidence from this study to support the research hypotheses. We will complete step 6 of hypothesis testing for this study in the next section, after we consider the calculations of the source table for a two-way between-groups ANOVA.

12-6: We calculate the total sum of squares using the following formula:

SS_total =∑(X − GM )².

We subtract the grand mean from every score to create deviations, then square the deviations, and finally sum the squared deviations.

12-7: We calculate the between-groups sum of squares for the first independent variable, that in the rows of the table of cells, using the following formula:

SS_{between(rows)} =∑(M_row − GM )².

For every participant, we subtract the grand mean from the marginal mean for the appropriate row for that participant. We square these deviations, and sum the squared deviations.

12-8: We calculate the between-groups sum of squares for the second independent variable, that in the columns of the table of cells, using the following formula:

SS_{between(columns)} =∑(M_column − GM )².

For every participant, we subtract the grand mean from the marginal mean for the appropriate column for that participant. We square these deviations, and then sum the squared deviations.

12-9: We calculate the within-groups sum of squares using the following formula:

SS_within =?∑(X − M_cell)².

For every participant, we subtract the appropriate cell mean from that participant’s score. We square these deviations, and sum the squared deviations.

12-10: To calculate the between-groups sum of squares for the interaction, we subtract the two between-groups sums of squares for the independent variables and the within-groups sum of squares from the total sum of squares. The formula is: SS_{between(interaction)} = SS_total − (SS_{between(rows)} + SS_{between(columns)} + SS_within).

12-11: The formulas to calculate the four mean squares are in Table 12-18. There are three between-groups mean squares—one for each main effect and one for the interaction—and one within-groups mean square. For each mean square, we divide the appropriate sum of squares by its related degrees of freedom. The formulas for the three F statistics, one for each main effect and one for the interaction, are also in Table 12-18. For each of the three effects, we divide the appropriate between-groups mean square by the within-group mean square. The denominator is the same in all three cases.

12-12: To calculate effect size for two-way ANOVA, we make three R² calculations, one for each main effect and one for the interaction. In each case, we divide the appropriate between-groups sum of squares by the total sum of squares minus the sums of squares for the other two effects. For example, the effect size for the interaction is calculated using this formula: R_interaction =

Let’s apply this to the ANOVA we just conducted. We use the statistics in the source table shown in Table 12-19 to calculate R² for each main effect and the interaction. Here are the calculations for the main effect for age:

Here are the calculations for the main effect for repetitions:

351

Here are the calculations for the interaction:

The conventions are the same as those presented in Chapter 11, shown again here in Table 12-20. From this table, we can see that the R² of 0.63 for the main effect of age and 0.52 for the interaction are very large. The R² of 0.00 for the main effect of repetitions indicates that there is no observable effect in this study.

Solutions to these Check Your Learning questions can be found in Appendix D.