In this section, we discuss two topics that are related to the procedures we have learned for inference about population means. First, we focus on an important issue when planning a study, specifically choosing the sample size. A wise user of statistics does not plan for inference without at the same time planning data collection. The second topic introduces us to various inference methods for non-Normal populations. These would be used when our populations are clearly non-Normal and we do not think that the sample size is large enough to rely on the robustness of the procedures.

Choosing the sample size

We describe sample size procedures for both confidence intervals and significance tests. For anyone planning to design a study, a general understanding of these procedures is necessary. While the actual formulas are a bit technical, statistical software now makes it trivial to get sample size results.

Sample size for confidence intervals

We can arrange to have both high confidence and a small margin of error by choosing an appropriate sample size. Let’s first focus on the one sample confidence interval. Its margin of error is

Besides the confidence level and sample size , this margin of error depends on the sample standard deviation . Because we don’t know the value of until we collect the data, we guess a value to use in the calculations. Thus, because is our estimate of the population standard deviation this value can also be considered our guess of the population standard deviation.

399

We will call this guessed value . We typically guess at this value using results from a pilot study or from similar studies published earlier. It is always better to use a value of the standard deviation that is a little larger than what is expected. This may result in a sample size that is a little larger than needed, but it helps avoid the situation where the resulting margin of error is larger than desired.

Given an estimate for and the desired margin of error , we can find the sample size by plugging everything into the margin of error formula and solving for . The one complication, however, is that depends not only on the confidence level but also on the sample size . Here are the details.

Finding the smallest sample size that satisfies this requirement can be done using the following iterative search:

Notice that this method makes no reference to the size of the population. It is the size of the sample that determines the margin of error. The size of the population does not influence the sample size we need as long as the population is much larger than the sample. Here is an example.

Note that the refers to the expected margin of error being no more than 15 minutes. This does not guarantee that the margin of error for the sample we collect will be less than 15 minutes. That is because the sample standard deviation varies sample to sample and these calculations are treating it as a fixed quantity. More advanced sample size procedures ask you to also specify the probability of obtaining a margin of error less than the desired value. For our approach, this probability is roughly 50%. For a probability closer to 100%, the sample size will need to be larger. For example, suppose we wanted this probability to be roughly 80%. In SAS, we’d perform these calculations using the command

proc power;

  onesamplemeans CI=t stddev=30 halfwidth=15

  probwidth=0.80 ntotal=.;

run;

The needed sample size increases from to .

401

Unfortunately, the actual number of usable observations is often less than that planned at the beginning of a study. This is particularly true of data collected in surveys or studies that involve a time commitment from the participants. Careful study designers often assume a nonresponse rate or dropout rate that specifies what proportion of the originally planned sample will fail to provide data. We use this information to calculate the sample size to be used at the start of the study. For example, if in the preceding survey we expect only 25% of those students to respond, we would need to start with a sample size of to obtain usable information from 18 students.

These sample size calculations also do not account for collection costs. In practice, taking observations costs time and money. There are times when the required sample size may be impossibly expensive. In those situations, one might consider a larger margin of error and/or a lower confidence level to be acceptable.

For the two-sample confidence interval, the margin of error is

A similar type of iterative search can be used to determine the sample sizes and , but now we need to guess both standard deviations and decide on an estimate for the degrees of freedom. We suggest taking the conservative approach and using the smaller of and for the degrees of freedom. Another approach is to consider the standard deviations and sample sizes are equal, so the margin of error is

and use degrees of freedom . That is the approach most statistical software take.

The power of the one-sample test

The power of a statistical test measures its ability to detect deviations from the null hypothesis. Because we usually hope to show that the null hypothesis is false, it is important to design a study with high power. Power calculations are a way to assess whether or not a sample size is sufficiently large to answer the research question.

The power of the one-sample test against a specific alternative value of the population mean is the probability that the test will reject the null hypothesis when this alternative is true. To calculate the power, we assume a fixed level of significance, usually .

Calculation of the exact power of the test takes into account the estimation of by and requires a new distribution. We will describe that calculation when discussing the power of the two-sample test. Fortunately, an approximate calculation that is based on assuming that is known is generally adequate for planning most studies in the one-sample case. This calculation is very much like that for the test, presented in Section 6.5. The steps are

Here is an example.

404

The power of the two-sample test

The two-sample test is one of the most used statistical procedures. Unfortunately, because of inadequate planning, users frequently fail to find evidence for the effects that they believe to be present. This is often the result of an inadequate sample size. Power calculations, performed prior to running the experiment, will help avoid this occurrence.

We just learned how to approximate the power of the one-sample test. The basic idea is the same for the two-sample case, but we will describe the exact method rather than an approximation again. The exact power calculation involves a new distribution, the noncentral distribution. This calculation is not practical by hand but is easy with software that calculates probabilities for this new distribution.

We consider only the common case where the null hypothesis is “no difference,” . We illustrate the calculation for the pooled two-sample test. A simple modification is needed when we do not pool. The unknown parameters in the pooled setting are , and a single common standard deviation . To find the power for the pooled two-sample test, follow these steps.

Step 1. Specify these quantities:

Step 2. Find the degrees of freedom and the value of that will lead to rejecting at your chosen level .

Step 3. Calculate the noncentrality parameter

Step 4. The power is the probability that a noncentral random variable with degrees of freedom df and noncentrality parameter will be greater than . Use software to calculate this probability. In SAS, the command is 1-PROBT(tstar, df,delta). If you do not have software that can perform this calculation, you can approximate the power as the probability that a standard Normal random variable is greater than , that is, . Use Table A or software for standard Normal probabilities.

405

Note that the denominator in the noncentrality parameter,

is our guess at the standard error for the difference in the sample means. Therefore, if we wanted to assess a possible study in terms of the margin of error for the estimated difference, we would examine times this quantity.

If we do not assume that the standard deviations are equal, we need to guess both standard deviations and then combine these to get an estimate of the standard error:

This guess is then used in the denominator of the noncentrality parameter. Use the conservative value, the smaller of and , for the degrees of freedom.

Inference for non-Normal populations

We have not discussed how to do inference about the mean of a clearly non-Normal distribution based on a small sample. If you face this problem, you should consult an expert. Three general strategies are available:

Each of these strategies quickly carries us beyond the basic practice of statistics. We emphasize procedures based on Normal distributions because they are the most common in practice, because their robustness makes them widely useful, and (most importantly) because we are first of all concerned with understanding the principles of inference.

407

Distribution-free significance tests do not require that the data follow any specific type of distribution such as Normal. This gain in generality isn’t free: if the data really are close to Normal, distribution-free tests have less power than tests. They also don’t quite answer the same question. The tests concern the population mean. Distribution-free tests ask about the population median, as is natural for distributions that may be skewed.

The sign test

The simplest distribution-free test, and one of the most useful, is the sign test. The test gets its name from the fact that we look only at the signs of the differences, not their actual values. The following example illustrates this test.

There are several varieties of the sign test, all based on counts and the binomial distribution. The sign test for matched pairs is the most useful. The null hypothesis of “no effect” is then always . The alternative can be one-sided in either direction or two-sided, depending on the type of change we are considering.

408

The matched pairs test in Example 7.7 tested the hypothesis that the mean of the distribution of differences is 0. The sign test in Example 7.20 is, in fact, testing the hypothesis that the median of the differences is 0. If is the probability that a difference is positive, then when the median is 0. This is true because the median of the distribution is the point with probability lying to its right. As Figure 7.15 illustrates, when the median is greater than 0, again because the probability to the right of the median is always . The sign test of against is a test of

The sign test in Example 7.20 makes no use of the actual scores—it just counts how many parts had a larger measurement with the option off. Any parts that did not have different measurements would be ignored altogether. Because the sign test uses so little of the available information, it is much less powerful than the test when the population is close to Normal. Chapter 16 describes other distribution-free tests that are more powerful than the sign test.