Simple linear regression studies the relationship between a response variable and an explanatory variable . We expect that different values of are associated with different mean responses for . We encountered a similar but simpler situation in Chapter 7 when we discussed methods for comparing two population means. Figure 10.1 illustrates a statistical model for comparing the items per hour entered by two groups of financial clerks using new data entry software. Group 2 received some training in the software while Group 1 did not. Entries per hour is the response variable. The treatment (training or not) is the explanatory variable. The model has two important parts:

Statistical model for simple linear regression

Now imagine giving different lengths of training to different groups of subjects. We can think of these groups as belonging to subpopulations, one for each possible value of . Each subpopulation consists of all individuals in the population having the same value of . If we gave hours of training to some subjects, hours of training to some others, and hours of training to some others, these three groups of subjects would be considered samples from the corresponding three subpopulations.

The statistical model for simple linear regression also assumes that, for each value of , the response variable is Normally distributed with a mean that depends on . We use to represent these means. In general, the means can change as changes according to any sort of pattern. In simple linear regression, we assume that the means all lie on a line when plotted against . To summarize, this model also has two important parts:

485

This statistical model is pictured in Figure 10.2. The line describes how the mean response changes with . This is the population regression line. The three Normal curves show how the response will vary for three different values of the explanatory variable . Each curve is centered at its mean response . All three curves have the same spread, measured by their common standard deviation .

From data analysis to inference

The data for a regression problem are the observed values of and . The model takes each to be a fixed known quantity, like the hours of training a worker has received.¹ The response for a given is a Normal random variable. The model describes the mean and standard deviation of this random variable. This model is not appropriate if there is error in measuring and it is large relative to the spread of the ’s. In these situations, more advanced inference methods are needed.

We use Case 10.1 to explain the fundamentals of simple linear regression. Because regression calculations in practice are always done by software, we rely on computer output for the arithmetic. Later in the chapter, we show formulas for doing the calculations. These formulas are useful in understanding analysis of variance (see Section 10.3) and multiple regression (see Chapter 11).

Let’s briefly review some of the ideas from Chapter 2 regarding least-squares regression. We start with a plot of the data, as in Figure 10.3, to verify that the relationship is approximately linear with no outliers. Always start with a graphical display of the data. There is no point in fitting a linear model if the relationship does not, at least approximately, appear linear. In this case, the distribution of income is skewed to the right (at each education level, there are many small incomes and just a few large incomes). Although the smoothed curve is roughly linear, the curve is being pulled toward the very large incomes, suggesting these observations could be influential.

A common remedy for a strongly skewed variable such as income is to consider transforming the variable prior to fitting a model. Here, the researchers considered the natural logarithm of income (LOGINC). Figure 10.4 is a scatterplot of these transformed data with a fitted smoothed curve in black and the least-squares regression line in green. The smoothed curve is almost linear, and the observations in the direction are more equally dispersed above and below this curve than the curve in Figure 10.3. Also, those four very large incomes no longer appear to be influential. Given these results, we continue our discussion of least-squares regression using the transformed data.

487

Recall that the least-squares line is the line that minimizes the sum of the squares of the residuals. The least-squares regression line also always passes through the point . These are helpful facts to remember when considering the fit of this line to a data set.

In Section 2.2 (pages 74–77), we discussed the correlation as a measure of association between two quantitative variables. In Section 2.3, we learned to interpret the square of the correlation as the fraction of the variation in that is explained by in a simple linear regression.

488

Having reviewed the basics of least-squares regression, we are now ready to proceed with a discussion of inference for regression. Here’s what is new in this chapter:

Our statistical model assumes that the responses are Normally distributed with a mean that depends upon in a linear way. Specifically, the population regression line

describes the relationship between the mean log income and the number of years of formal education in the population. The slope is the mean increase in log income for each additional year of education. The intercept is the mean log income when an entrepreneur has years of formal education. This parameter, by itself, is not meaningful in this example because years of education would be extremely rare.

Because the means lie on the line , they are all determined by and . Thus, once we have estimates of and , the linear relationship determines the estimates of for all values of . Linear regression allows us to do inference not only for subpopulations for which we have data, but also for those corresponding to ’s not present in the data. These -values can be both within and outside the range of observed ’s. However, extreme caution must be taken when performing inference for an -value outside the range of the observed ’s because there is no assurance that the same linear relationship between and holds.

We cannot observe the population regression line because the observed responses vary about their means. In Figure 10.4 we see the least-squares regression line that describes the overall pattern of the data, along with the scatter of individual points about this line. The statistical model for linear regression makes the same distinction. This was displayed in Figure 10.2 with the line and three Normal curves. The population regression line describes the on-the-average relationship and the Normal curves describe the variability in for each value of .

489

Think of the model in the form

The FIT part of the model consists of the subpopulation means, given by the expression . The RESIDUAL part represents deviations of the data from the line of population means. The model assumes that these deviations are Normally distributed with standard deviation . We use (the lowercase Greek letter epsilon) to stand for the RESIDUAL part of the statistical model. A response is the sum of its mean and a chance deviation from the mean. The deviations represent “noise,” variation in due to other causes that prevent the observed -values from forming a perfectly straight line on the scatterplot.

The simple linear regression model can be justified in a wide variety of circumstances. Sometimes, we observe the values of two variables, and we formulate a model with one of these as the response variable and the other as the explanatory variable. This was the setting for Case 10.1, where the response variable was log income and the explanatory variable was the number of years of formal education. In other settings, the values of the explanatory variable are chosen by the persons designing the study. The scenario illustrated by Figure 10.2 is an example of this setting. Here, the explanatory variable is training time, which is set at a few carefully selected values. The response variable is the number of entries per hour.

For the simple linear regression model to be valid, one essential assumption is that the relationship between the means of the response variable for the different values of the explanatory variable is approximately linear. This is the FIT part of the model. Another essential assumption concerns the RESIDUAL part of the model. The assumption states that the residuals are an SRS from a Normal distribution with mean zero and standard deviation . If the data are collected through some sort of random sampling, this assumption is often easy to justify. This is the case in our two scenarios, in which both variables are observed in a random sample from a population or the response variable is measured at predetermined values of the explanatory variable.

In many other settings, particularly in business applications, we analyze all of the data available and there is no random sampling. Here, we often justify the use of inference for simple linear regression by viewing the data as coming from some sort of process. The line gives a good description of the relationship, the fit, and we model the deviations from the fit, the residuals, as coming from a Normal distribution.

490

Estimating the regression parameters

The method of least squares presented in Chapter 2 fits the least-squares line to summarize a relationship between the observed values of an explanatory variable and a response variable. Now we want to use this line as a basis for inference about a population from which our observations are a sample. We can do this only when the statistical model for regression is reasonable. In that setting, the slope and intercept of the least-squares line

estimate the slope and the intercept of the population regression line.

491

Recalling the formulas from Chapter 2, the slope of the least-squares line is

and the intercept is

Here, is the correlation between the observed values of and , is the standard deviation of the sample of ’s, and is the standard deviation of the sample of ’s. Notice that if the estimated slope is 0, so is the correlation, and vice versa. We discuss this relationship more later in this chapter.

The remaining parameter to be estimated is , which measures the variation of about the population regression line. More precisely, is the standard deviation of the Normal distribution of the deviations in the regression model. However, we don’t observe these , so how can we estimate ?

Recall that the vertical deviations of the points in a scatterplot from the fitted regression line are the residuals. We use for the residual of the th observation:

The residuals are the observable quantities that correspond to the unobservable model deviations . The sum to 0, and the come from a population with mean 0. Because we do not observe the , we use the residuals to estimate and check the model assumptions of the .

To estimate , we work first with the variance and take the square root to obtain the standard deviation. For simple linear regression the estimate of is the average squared residual

We average by dividing the sum by in order to make an unbiased estimator of . The sample variance of observations use the divisor for the same reason. The residuals are not separate quantities. When any residuals are known, we can find the other two. The quantity is the degrees of freedom of . The estimate of the model standard deviation is given by

We call the regression standard error.

492

In practice, we use software to calculate , , and from data on and . Here are the results for the income example of Case 10.1.

Conditions for regression inference

You can fit a least-squares line to any set of explanatory-response data when both variables are quantitative. The simple linear regression model, which is the basis for inference, imposes several conditions on this fit. We should always verify these conditions before proceeding to inference. There is no point in trying to do statistical inference if the data do not, at least approximately, meet the conditions that are the foundation for the inference.

The conditions concern the population, but we can observe only our sample. Thus, in doing inference, we act as if the sample is an SRS from the population. For the study described in Case 10.1, the researchers used a national survey. Participants were chosen to be a representative sample of the United States, so we can treat this sample as an SRS. The potential for bias should always be considered, especially when obtaining volunteers.

The next condition is that there is a linear relationship in the population, described by the population regression line. We can’t observe the population line, so we check this condition by asking if the sample data show a roughly linear pattern in a scatterplot. We also check for any outliers or influential observations that could affect the least-squares fit. The model also says that the standard deviation of the responses about the population line is the same for all values of the explanatory variable. In practice, the spread of observations above and below the least-squares line should be roughly the same as varies.

Plotting the residuals against the explanatory variable or against the predicted (or fitted) values is a helpful and frequently used visual aid to check these conditions. This is better than the scatterplot because a residual plot magnifies patterns. The residual plot in Figure 10.7 for the data of Case 10.1 looks satisfactory. There is no curved pattern or data points that seem out of the ordinary, and the data appear equally dispersed above and below zero throughout the range of .

The final condition is that the response varies Normally about the population regression line. In that case, we expect the residuals to also be Normally distributed.⁴ A Normal quantile plot of the residuals (Figure 10.8) shows no serious deviations from a Normal distribution. The data give no reason to doubt the simple linear regression model, so we proceed to inference.

There is no condition that requires Normality for the distributions of the response or explanatory variables. The Normality condition applies only to the distribution of the model deviations, which we assess using the residuals. For the entrepreneur problem, we transformed to get a more linear relationship as well as residuals that appear Normal with constant variance. The fact that the marginal distribution of the transformed is more Normal is purely a coincidence.

495

Confidence intervals and significance tests

Chapter 7 presented confidence intervals and significance tests for means and differences in means. In each case, inference rested on the standard errors of estimates and on distributions. Inference for the slope and intercept in linear regression is similar in principle. For example, the confidence intervals have the form

where is a critical value of a distribution. It is the formulas for the estimate and standard error that are different.

496

Confidence intervals and tests for the slope and intercept are based on the sampling distributions of the estimates and . Here are some important facts about these sampling distributions:

Normality of and is a consequence of Normality of the individual deviations in the regression model. If the are not Normal, a general form of the central limit theorem tells us that the distributions of and will be approximately Normal when we have a large sample. Regression inference is robust against moderate lack of Normality. On the other hand, outliers and influential observations can invalidate the results of inference for regression.

Because and have Normal sampling distributions, standardizing these estimates gives standard Normal statistics. The standard deviations of these estimates are multiples of . Because we do not know , we estimate it by , the variability of the data about the least-squares line. When we do this, we get distributions with degrees of freedom , the degrees of freedom of . We give formulas for the standard errors and in Section 10.3. For now, we concentrate on the basic ideas and let software do the calculations.

497

Formulas for confidence intervals and significance tests for the intercept are exactly the same, replacing and by and its standard error . Although computer outputs often include a test of , this information usually has little practical value. From the equation for the population regression line, , we see that is the mean response corresponding to . In many practical situations, this subpopulation does not exist or is not interesting.

On the other hand, the test of is quite useful. When we substitute in the model, the term drops out and we are left with

This model says that the mean of does not vary with . In other words, all the come from a single population with mean , which we would estimate by . The hypothesis , therefore, says that there is no straight-line relationship between and and that linear regression of on is of no value for predicting .

498

In this example, we can discuss percent change in income for a unit change in education because the response variable is on the log scale and is not. In business and economics, we often encounter models in which both variables are on the log scale. In these cases, the slope approximates the percent change in for a 1% change in . This is known as elasticity, which is a very important concept in economic theory.

500

The word “regression”

To “regress” means to go backward. Why are statistical methods for predicting a response from an explanatory variable called “regression”? Sir Francis Galton (1822–1911) was the first to apply regression to biological and psychological data. He looked at examples such as the heights of children versus the heights of their parents. He found that the taller-than-average parents tended to have children who were also taller than average, but not as tall as their parents. Galton called this fact “regression toward mediocrity,” and the name came to be applied to the statistical method. Galton also invented the correlation coefficient and named it “correlation.”

Why are the children of tall parents shorter on the average than their parents? The parents are tall in part because of their genes. But they are also tall in part by chance. Looking at tall parents selects those in whom chance produced height. Their children inherit their genes, but not their good luck. As a group, the children are taller than average (genes), but their heights vary by chance about the average, some upward and some downward. The children, unlike the parents, were not selected because they were tall and thus, on average, are shorter. A similar argument can be used to describe why children of short parents tend to be taller than their parents.

Here’s another example. Students who score at the top on the first exam in a course are likely to do less well on the second exam. Does this show that they stopped studying? No—they scored high in part because they knew the material but also in part because they were lucky. On the second exam, they may still know the material but be less lucky. As a group, they will still do better than average but not as well as they did on the first exam. The students at the bottom on the first exam will tend to move up on the second exam, for the same reason.

The regression fallacy is the assertion that regression toward the mean shows that there is some systematic effect at work: students with top scores now work less hard, or managers of last year’s best-performing mutual funds lose their touch this year, or heights get less variable with each passing generation as tall parents have shorter children and short parents have taller children. The Nobel economist Milton Friedman says, “I suspect that the regression fallacy is the most common fallacy in the statistical analysis of economic data.”⁶ Beware.

Inference about correlation

The correlation between log income and level of education for the 100 entrepreneurs is . This value appears in the Excel output in Figure 10.5 (page 492), where it is labeled “Multiple R.”⁷ We might expect a positive correlation between these two measures in the population of all entrepreneurs in the United States. Is the sample result convincing evidence that this is true?

This question concerns a new population parameter, the population correlation. This is the correlation between the log income and level of education when we measure these variables for every member of the population. We call the population correlation , the Greek letter rho. To assess the evidence that in the population, we must test the hypotheses

501

It is natural to base the test on the sample correlation . Table G in the back of the book shows the one-sided critical values of . To use software for the test, we exploit the close link between correlation and the regression slope. The population correlation is zero, positive, or negative exactly when the slope of the population regression line is zero, positive, or negative. In fact, the statistic for testing also tests . What is more, this statistic can be written in terms of the sample correlation .

502

The alternative formula for the test statistic is convenient because it uses only the sample correlation and the sample size . Remember that correlation, unlike regression, does not require the distinction between explanatory and response variables. For variables and , there are two regressions ( on and on ) but just one correlation. Both regressions produce the same statistic.

The distinction between the regression setting and correlation is important only for understanding the conditions under which the test for 0 population correlation makes sense. In the regression model, we take the values of the explanatory variable as given. The values of the response are Normal random variables, with means that are a straight-line function of . In the model for testing correlation, we think of the setting where we obtain a random sample from a population and measure both and . Both are assumed to be Normal random variables. In fact, they are taken to be jointly Normal. This implies that the conditional distribution of for each possible value of is Normal, just as in the regression model.