Chapter 1. Statistical Significance

Statistical Applets

To set up the test, fill in the boxes: What null hypothesis H0 about the mean μ do you want to test? Which alternative hypothesis Ha do you have in mind, and what level of significance α do you require? What value of the standard deviation σ is known to be true? How many observations n will you have (250 or fewer)?

If you already have a sample mean, enter this value and click UPDATE to display the sample mean on the graph and calculate its statistical significance. Or you can specify the true population mean μ to create a random sample from the population, display the observations and sample mean (note that some of the points in the sample may be too far from μ to appear in the display), and calculate its statistical significance. Click NEW SAMPLE to generate different random samples.

This applet illustrates statistical tests with fixed level of significance. Here we're testing a hypothesis about the mean of a normal distribution whose standard deviation we know, but the concepts are essentially the same for any other type of significance test.

The normal curve shows the sampling distribution of the sample mean when your null hypothesis is true and the sample size and population standard deviation have the values you choose. The yellow area under the curve represents the values of that are significant at level α. This area is equal to α, the probability of getting a significant result when the null hypothesis is true.

1.

Suppose you're planning to collect a set of data in an experiment where the null hypothesis states that the population mean will be 15. You will collect 30 observations, and you expect the population standard deviation to be 6.5. You plan to test at the .05 level of significance, using a one-tailed test (that is, testing whether μ < 15). Below, enter the values you would enter in the applet to simulate this situation:

  • H0: μ =
  • α =
  • σ =
  • n =
2
Try again.
Incorrect. See above for the correct answers.
Great job.

2.

Suppose that you suspect that the observations in your experiment will actually be centered around µ = 13. If this is the case, then what sample mean would you expect to observe for your 30 observations?
2
Correct.
Try again.
Incorrect.

3.

Choose the "I have data, and the observed..." option in the applet, enter this value (13), and click UPDATE.

  • According to the applet, what would be the P-value for your experiment if you observe a sample mean of 13?
  • Would you be able to reject the null hypothesis in this situation?
2
Try again.
Incorrect. See above for the correct answers.
Great job.

4.

Now choose the option in the applet that states "The truth about the population is...", fill in your expected value for µ (13), and generate 10 new samples. For each sample, note whether the P-value for the sample is low enough to reject the null hypothesis. In how many samples would you be able to reject the null hypothesis? Did the results of this exercise surprise you? Why or why not?
You probably observed that, even given a value for the true population mean where the null hypothesis "should" be rejected, actual samples of data will often fail to allow you to reject the null hypothesis. If you think this through, though, it should not be all that surprising. In the situation we've set up here, P = .046 when the sample mean is exactly equal to the population mean, 13. This is just on the cusp of statistical significance. When you take samples from this population mean, half of the samples should come out with sample means lower than the population mean, meaning that the null hypothesis would be rejected. But half the samples should also come out with sample means higher than the population mean, meaning that the null hypothesis would fail to be rejected. As this example clearly illustrates, when random variability is involved (meaning, in just about every situation you will encounter) you have to be very careful about how you plan and interpret results of experiments.