The confidence interval is appropriate when our goal is to estimate population parameters. The second common type of inference is directed at a quite different goal: to assess the evidence provided by the data in favor of some claim about the population parameters.

The reasoning of significance tests

A significance test is a formal procedure for comparing observed data with a hypothesis whose truth we want to assess. The hypothesis is a statement about the parameters in a population or model. The results of a test are expressed in terms of a probability that measures how well the data and the hypothesis agree. We use the following Case Study and subsequent examples to illustrate these ideas.

317

Notice that in Case 6.1, there is an intimate understanding of what is important to be discovered. In particular, is the population mean too high or too low relative to a desired level? With an understanding of what role the data play in the discovery process, we are able to formulate appropriate hypotheses. If the bottler were concerned only about the possible underfilling of bottles, then the hypotheses of interest would change. Let us proceed with the question of whether the bottling process is either underfilling or overfilling bottles.

318

What are the key steps in this example?

The probability is quite small. Something that happens with probability 0.00058 occurs only about six times out of 10,000. In this case, we have two possible explanations:

Because this probability is so small, we prefer the second conclusion: the process mean is not 473 ml. It should be emphasized that to “conclude” does not mean we know the truth or that we are right. There is always a chance that our conclusion is wrong. Always bear in mind that when dealing with data, there are no guarantees. We now turn to an example in which the data suggest a different conclusion.

The probabilities in Examples 6.13 and 6.14 are measures the compatibility of the data (sample means of 474.54 and 472.56) with the null hypothesis that . Figure 6.12 compares these two results graphically. The Normal curve is the sampling distribution of when . You can see that we are not particularly surprised to observe , but is clearly an unusual data result. Herein lies the core reasoning of statistical tests: a data result that is extreme if a hypothesis were true is evidence that the hypothesis may not be true. We now consider some of the formal aspects of significance testing.

319

Stating hypotheses

In Examples 6.13 and 6.14, we asked whether the fill data are plausible if, in fact, the true mean fill amount for all bottles ( ) is 473 ml. That is, we ask if the data provide evidence against the claim that the population mean is 473. The first step in a test of significance is to state a claim that we will try to find evidence against.

A null hypothesis is a statement about the population or process parameters. For example, the null hypothesis for Examples 6.13 and 6.14 is

Note that the null hypothesis refers to the process mean for all filled bottles, including those we do not have data on.

It is convenient also to give a name to the statement that we hope or suspect is true instead of . This is called the alternative hypothesis and is abbreviated as . In Examples 6.13 and 6.14, the alternative hypothesis states that the mean fill amount is not 473. We write this as

Hypotheses always refer to some population, process, or model, not to a particular data outcome. For this reason, we always state and in terms of population parameters.

Because expresses the effect that we hope to find evidence for, we will sometimes begin with and then set up as the statement that the hoped-for effect is not present. Stating , however, is often the more diffcult task. It is not always clear, in particular, whether should be one-sided or two-sided, which refers to whether a parameter differs from its null hypothesis value in a specific direction or in either direction.

320

The alternative in the bottle-filling examples is two-sided. In both examples, we simply asked if mean fill amount is off target. The process can be off target in that it fills too much or too little on average, so we include both possibilities in the alternative hypothesis. Here, the alternative is not a good situation in the sense that the process mean is off target. Thus, it is not our hope the alternative is true. It, however, is our hope that we can detect when the process has gone off target so that corrective actions can be done. Here is a setting in which a one-sided alternative is appropriate.

The alternative hypothesis should express the hopes or suspicions we bring to the data. It is cheating to first look at the data and then frame to fit what the data show. If you do not have a specific direction firmly in mind in advance, you must use a two-sided alternative. Moreover, some users of statistics argue that we should always use a two-sided alternative.

The choice of the hypotheses in Example 6.15 as

deserves a final comment. We do not expect that elimination of steps in order processing would actually increase the processing time. However, we can allow for an increase by including this case in the null hypothesis. Then we would write

This statement is logically satisfying because the hypotheses account for all possible values of . However, only the parameter value in that is closest to influences the form of the test in all common significance-testing situations. Think of it this way: if the data lead us away from to believing that , then the data would certainly lead us away from believing that because this involves values of that are in the opposite direction to which the data are pointing. Moving forward, we take to be the simpler statement that the parameter equals a specific value, in this case .

Test statistics

We learn the form of significance tests in a number of common situations. Here are some principles that apply to most tests and that help in understanding the form of tests:

A test statistic measures compatibility between the null hypothesis and the data. We use it for the probability calculation that we need for our test of significance. It is a random variable with a distribution that we know.

Let’s return to our bottle filling example and specify the hypotheses as well as calculate the test test statistic.

322

As stated in Example 6.13, past production shows that the fill amounts of the individual bottles are not too far from the Normal distribution. In that light, we can be confident enough that, with a sample size of 20, the distribution of the sample is close enough to the Normal for working purposes. In turn, the standardized test statistic will have approximately the distribution. We use facts about the Normal distribution in what follows.

-values

If all test statistics were Normal, we could base our conclusions on the value of the test statistic. In fact, the Supreme Court of the United States has said that “two or three standard deviations” ( is its criterion for rejecting (see Exercise 6.59, page 326), and this is the criterion used in most applications involving the law. But because not all test statistics are Normal, as we learn in subsequent chapters, we use the language of probability to express the meaning of a test statistic.

A test of significance finds the probability of getting an outcome as extreme or more extreme than the actually observed outcome. “Extreme” means “far from what we would expect if were true.” The direction or directions that count as “far from what we would expect” are determined by and .

The key to calculating the -value is the sampling distribution of the test statistic. For the problems we consider in this chapter, we need only the standard Normal distribution for the test statistic .

Statistical significance

We started our discussion of significance tests with the statement of null and alternative hypotheses. We then learned that a test statistic is the tool used to examine the compatibility of the observed data with the null hypothesis. Finally, we translated the test statistic into a -value to quantify the evidence against . One important final step is needed: to state our conclusion.

We can compare the -value we calculated with a fixed value that we regard as decisive. This amounts to announcing in advance how much evidence against we will require to reject . The decisive value is called the significance level. It is commonly denoted by (the Greek letter alpha). If we choose , we are requiring that the data give evidence against so strong that it would happen no more than 5% of the time (1 time in 20) when is true. If we choose , we are insisting on stronger evidence against , evidence so strong that it would appear only 1% of the time (1 time in 100) if is, in fact, true.

“Signifcant” in the statistical sense does not mean “important.” The original meaning of the word is “signifying something.” In statistics, the term is used to indicate only that the evidence against the null hypothesis has reached the standard set by . For example, significance at level 0.01 is often expressed by the statement “The results were signifcant .” Here stands for the -value. The -value is more informative than a statement of significance because we can then assess significance at any level we choose. For example, a result with is signifcant at the level but is not signifcant at the level. We discuss this in more detail at the end of this section.

Note that the calculated -value for this example is actually 0.0006, but we reported the result as . The value 0.001, 1 in 1000, is sufficiently small to force a clear rejection of . When encountering a very small -value as in Example 6.19, standard practice is to provide the test statistic value and report the -value as simply less than 0.001.

Examples 6.16 through 6.19 in sequence showed us that a test of significance is a process for assessing the significance of the evidence provided by the data against a null hypothesis. These steps provide the general template for all tests of significance. Here is a general summary of the four common steps.

We learn the details of many tests of significance in the following chapters. The proper test statistic is determined by the hypotheses and the data collection design. We use computer software or a calculator to find its numerical value and the -value. The computer will not formulate your hypotheses for you, however. Nor will it decide if significance testing is appropriate or help you to interpret the -value that it presents to you. These steps require judgment based on a sound understanding of this type of inference.

Tests of one population mean

We have noted the four steps common to all tests of significance. We have also illustrated these steps with the bottle filling scenario of Case 6.1 (page 317). Here is a summary for the test of one population mean.

We want to test a population parameter against a specified value. This is the null hypothesis. For a test of a population mean , the null hypothesis is

which often is expressed as

where is the hypothesized value of that we would like to examine.

The test is based on data summarized as an estimate of the parameter. For a population mean, this is the sample mean . Our test statistic measures the difference between the sample estimate and the hypothesized parameter in terms of standard deviations of the test statistic:

Recall from Section 6.1 that the standard deviation of is . Therefore, the test statistic is

Again recall from Section 6.1 that, if the population is Normal, then will be Normal and will have the standard Normal distribution when is true. By the central limit theorem, both distributions will be approximately Normal when the sample size is large, even if the population is not Normal. We assume that we’re in one of these two settings for now.

Suppose that we have calculated a test statistic . If the alternative is one-sided on the high side, then the -value is the probability that a standard Normal random variable takes a value as large or larger than the observed 1.7. That is,

Similar reasoning applies when the alternative hypothesis states that the true lies below the hypothesized (one-sided). When states that is simply unequal to (two-sided), values of away from zero in either direction count against the null hypothesis. The -value is the probability that a standard Normal is at least as far from zero as the observed . Again, if the test statistic is , the two-sided -value is the probability that or . Because the standard Normal distribution is symmetric, we calculate this probability by finding and doubling it:

327

We would make exactly the same calculation if we observed . It is the absolute value that matters, not whether is positive or negative. Here is a statement of the test in general terms.

The reported statement does not imply that we conclude that the null hypothesis is true, only that the level of evidence we require to reject the null hypothesis is not met. Our criminal court system follows a similar procedure in which a defendant is presumed innocent until proven guilty. If the level of evidence presented is not strong enough for the jury to find the defendant guilty beyond a reasonable doubt, the defendant is acquitted. Acquittal does not imply innocence, only that the degree of evidence was not strong enough to prove guilt.

Two-sided significance tests and confidence intervals

Recall the basic idea of a confidence interval, discussed in Section 6.2. We constructed an interval that would include the true value of with a specified probability . Suppose that we use a 95% confidence interval . Then the values of that are not in our interval would seem to be incompatible with the data. This sounds like a significance test with (or 5%) as our standard for drawing a conclusion. The following example demonstrates that this is correct.

330

First, the test. The mean of the sample is . Given the large sample size of , it is fairly safe to use the reported standard deviation of 80.6% as . The test statistic is

Because the alternative is two-sided, the -value is

The largest value of in Table A is 3.49. Even though we cannot determine the exact probability from the table, it is pretty obvious that the -value is much less than 0.001. There is overwhelming evidence that the mean initial return for the Chinese IPO population is not 0.

To compute a 99% confidence interval for the mean IPO initial return, find in Table D the critical value for 99% confidence. It is , the same critical value that marked off signifcant ’s in our test. The confidence interval is

The hypothesized value falls well outside this confidence interval. In other words, it is in the region we are 99% confident is not in. Thus, we can reject

at the 1% significance level. However, we might want to test the Chinese market against other markets. For example, certain IPO markets, such as technology and the “dot-com” markets, have shown abnormally high initial returns. Suppose we wish to test the Chinese market against a market that has value of 65. Because the value of 65 lies inside the 99% confidence interval for , we cannot reject

Figure 6.15 illustrates both cases.

The calculation in Example 6.21 for a 1% significance test is very similar to the calculation for a 99% confidence interval. In fact, a two-sided test at significance level can be carried out directly from a confidence interval with confidence level .

331

-values versus reject-or-not reporting

Imagine that we are conducting a two-sided test and find the observed to be 2.41. Suppose we have picked a significance level of . We can find from Table A or the bottom of Table D, that a value gives us a point on the standard Normal distribution such that 5% of the distribution is beyond ±1.96. Given , we would reject the null hypothesis for . We take the absolute value of the observed because had we gotten a of -2.41, we would need to arrive at the same conclusion of rejection. For one-sided testing with , we would compare our observed with for a less-than alternative and with greater-than alternative.

A value of that is used to compare the observed against is called a critical value. From our preceding discussion, we could report, “With an observed statistic of 2.41, the data lead us to reject the null hypothesis at the 5% level of significance.” What if the reader of the report sets his or her bar at the 1% level of significance? What would the conclusion of our report be now? The way our report presently stands, we would be forcing the reader to find out for themselves the conclusion at the 1% level. It would be even worse if we did not report the observed value and simply reported that the results are signifcant at the 5% level . It is equally noninformative to report that the results are insignifcant at the 5% level . Clearly, these examples of significance test reporting are very self limiting.

Consider now the reporting of the -value as we have done with all our examples. For the two-sided alternative and an observed of 2.41, the -value is

The -value gives a better sense of how strong the evidence is. Notice how much more informative and convenient for others if we report, “The data lead us to reject the null hypothesis (, ).” Namely, we find the result is signifcant at the level because . But, it is not signifcant at the level because the -value is larger than 0.01. From Figure 6.16, we see that the P-value is the smallest level a at which the data are signifcant. With -value in hand, we don’t need to search tables to find different critical values to compare against for different values of . Knowing the -value allows us, or anyone else, to assess sig-nifcance at any level with ease. With this said, the -value is not the “answer all” of a statistical study. As will be emphasized in Section 6.4, a result that is found to be statistically signifcant does not necessarily imply practically important.

332

Our discussion clearly encourages the reporting of -values as opposed to the reject-or-not reporting based on some fixed such as 0.05. The practice of statistics almost always employs computer software or a calculator that calculates -values automatically. In practice, the use of tables of critical values is becoming outdated. Notwithstanding, we include the usual tables of critical values (such as Table D) at the end of the book for learning purposes and to rescue students without computing resources.