Chapter 102. Statistical Significance

1. After researchers have collected data, they typically calculate descriptive statistics, such as the mean and standard deviation. These statistics provide a summary of the central tendency and variability of each distribution of scores. Next, they calculate inferential statistics to determine whether the results support their hypotheses about behavior. For example, if researchers are comparing an experimental group with a control group, the descriptive statistics indicate whether or not the experimental treatment changed performance, and the inferential statistics indicate how much confidence the researchers should have in the results.

2. The questions that inferential statistics can answer about a hypothesis usually involve a comparison. Did one group of participants perform differently than another group? On average, did participants perform differently in one condition of an experiment than in other conditions? Does the correlation coefficient for two variables differ from zero? (A correlation of zero indicates no relationship.)

3. If researchers find a difference among groups of participants, they want to know whether or not it is a “real” difference—a reliable, repeatable result—rather than a fluke due to random errors in measurement, random fluctuations in people’s behavior, or failure of random assignment to control for preexisting differences between the groups. Roughly speaking, we consider a result to be statistically significant if there is a low probability that the result was due to chance factors.

4. One standard method of determining statistical significance is to pose a null hypothesis—typically claiming that there is no difference—and then testing the likelihood that the observed results would have occurred if the null hypothesis were true. This method is called null hypothesis significance testing (NHST). For example, if researchers are comparing an experimental group to a control group, the null hypothesis is that the two groups do not differ. Assuming that the null hypothesis is true, the likelihood of obtaining results equal to or greater than the observed results is called the probability level, or p-level. Typically, if the p-level is less than 0.05 (5 percent), the result is considered statistically significant.

5. There is a long-standing controversy within psychology as to whether the NHST approach is an appropriate way to evaluate research results. The p-level takes into account the size of the sample; the larger the sample, the easier it is to obtain a statistically significant result (with p < .05), even with a very small difference between groups. Also, the p-level is frequently misinterpreted as indicating the probability that the null hypothesis is true. That is actually backwards. Given 100 percent probability that the null hypothesis is true, the p-level indicates the probability of obtaining a difference equal to or greater than the difference measured (observed in the results).

6. In addition to calculating the p-level, researchers usually calculate the effect size of a difference, which indicates the magnitude of the difference in a way that is not influenced by the size of the sample. Instead of using the NHST approach, some researchers prefer to calculate the confidence interval (margin of error) for each statistic, and then use those numbers to evaluate the reliability and importance of the finding. Researchers typically calculate the confidence interval for a mean in a way that provides a 95 percent probability that the actual mean of the population falls within the interval.

7. On a graph of the research results, the mean scores often have error bars indicating the 95 percent confidence intervals. This allows the viewer to estimate the likelihood that the observed difference is a meaningful difference. In the left graph, the confidence intervals for females and males overlap, suggesting that the difference in mean scores is not reliable. In the right graph, the non-overlapping confidence intervals suggest that the measured age gap in performance could be a real difference.

8. Finally, it is important to remember that a statistically significant result (with p < .05) is not necessarily an important result, even if the p-level is very small (such as p < .01 or p < .001). A small difference may be reliable and repeatable, but not have much impact on people’s behavior. For example, in the study illustrated in this graph, headache sufferers who had a pain level of 10 experienced a 40 percent decrease in pain after taking one aspirin tablet. Those who took two tablets experienced a 46 percent improvement. The difference was statistically significant, but most people would not notice the difference in their daily life.