Statistical model for linear regression

Simple linear regression studies the relationship between a response variable y and a single explanatory variable x. We expect that different values of x will produce different mean responses for y. We encountered a similar but simpler situation in Chapter 7 when we discussed methods for comparing two population means. Figure 10.1 illustrates the statistical model for a comparison of blood pressure change in two groups of experimental subjects. Group 2 subjects were provided extra servings of fruits and vegetables in a calorie-controlled diet, while Group 1 subjects were not. We can think of the extra servings of fruits and vegetables (yes or no) as the explanatory variable in this example. This model has two important parts:

In linear regression, the explanatory variable x is quantitative and can have many different values. Imagine, for example, giving a different number of servings of fruits and vegetables x to different groups of college students. We can think of the values of x as defining different subpopulationssubpopulations, one for each possible value of x. Each subpopulation consists of all individuals in the college student population having the same value of x. If we gave x = 5 servings to some students, x = 7 servings to some others, x = 9 servings to some others, and x = 11 servings to the rest, these four groups of students would be considered samples from the corresponding four subpopulations.

The statistical model for simple linear regression assumes that for each value of x, the observed values of the response variable y are Normally distributed with a mean that depends on x. We use μ_y to represent these means. In general, the means μ_y can change as x changes according to any sort of pattern. In simple linear regressionsimple linear regression, the means all lie on a line when plotted against x. To summarize, this model also has two important parts:

The simple linear regression model is pictured in Figure 10.2. The line describes how the mean response μ_y changes with x. This is the population regression linepopulation regression line. The four Normal curves show how the response y will vary for four different values of the explanatory variable x. Each curve is centered at its mean response μ_y. All four curves have the same spread, measured by their common standard deviation σ.

558

Preliminary data analysis and inference considerations

The data for a linear regression are observed values of y and x. The model takes each x to be a known quantity, like the number of servings of fruits and vegetables. The response y for a given x is assumed to be a Normal random variable. The linear regression model describes the mean and standard deviation of this random variable.

We use the following example to explain the fundamentals of simple linear regression. Because regression calculations in practice are done by statistical software, we rely on computer output for the arithmetic. In Section 10.2, we give an example that illustrates how to do the work with a calculator or spreadsheet.

Before starting our analysis, it is appropriate to consider the extent to which the results can reasonably be generalized. In the original study, undergraduate volunteers were obtained at a large southeastern public university through classroom announcements and campus flyers. The potential for bias should always be considered when obtaining volunteers. In this case, the participants were screened, and those with severe health issues, as well as varsity athletes, were excluded. As a result, the researchers considered these volunteers as a simple random sample (SRS) from the population of undergraduates at this university. However, they acknowledged the risks of generalizing further, stating that similar investigations at universities of different sizes and in other climates of the United States are needed.

559

Another issue to consider in this example is the fact that the explanatory variable is not exactly known but, instead, estimated using a FitBit Flex worn over a one-week period. If there is error in measuring x and it is large relative to the spread of the x’s, more advanced inference methods are needed. In this case, the measurement error is expected to be relatively small. Subjects were instructed to continue normal activities over the one-week period and to notify the researchers of any deviations. Also, this estimate of physical activity represents an average over seven days.

Always start with a graphical display of the data. There is no point in fitting a linear model if the relationship is not approximately linear. A graphical display can also be used to assess the direction and strength of the relationship and to identify outliers and influential observations.

In this example, subpopulations are defined by the explanatory variable, physical activity. The considerable amount of scatter about the least-squares regression line suggests a large amount of variation of BMI in each subpopulation. Why might this occur? Consider sampling women from your university, each averaging the same number of steps per day—say, 9000. Even though the average number of steps per day is the same, you would not expect all these women to have the same BMI. Variation in other factors such as genetic makeup, lifestyle, and diet should all contribute to the variation of BMI.

The statistical model for linear regression assumes that these BMI values are Normally distributed with a mean μ_y that depends upon x in a linear way. Specifically,

μ_y = β₀ + β₁x

560

This was displayed in Figure 10.2 with the line and the four Normal curves. The line is the population regression line, which gives the average BMI for all values of x. The Normal curves provide a description of the variation of BMI about these means. The following equation expresses this idea:

DATA = FIT + RESIDUAL

The FIT part of the model consists of the subpopulation means, given by the expression β₀ + β₁x. The RESIDUAL part represents deviations of the data from the line of population means. We assume that these deviations are Normally distributed with mean 0 and standard deviation σ.

We use ϵ (the lowercase Greek letter epsilon) to stand for the RESIDUAL part of the statistical model. A response y is the sum of its mean and a chance deviation ϵ from the mean. These model deviations ϵ represent “noise,’’ that is, variation in y due to other causes that prevent the observed (x, y)-values from forming a perfectly straight line on the scatterplot.

561

Estimating the regression parameters

The method of least squares presented in Chapter 2 fits a line to summarize a relationship between the observed values of an explanatory variable and a response variable. Now we want to use the least-squares line as a basis for inference about a population from which our observations are a sample. In that setting, the slope b₁ and intercept b₀ of the least-squares line

estimate the slope β₁ and the intercept β₀ of the population regression line.

This inference should only be done when the statistical model assumptions just presented are reasonably met. Model checks are needed and some judgment is required. Because additional methods to check model assumptions rely on first fitting the model to the data, let’s briefly review the methods of Chapter 2 concerning least-squares regression.

Using the formulas from Chapter 2 (page 112), the slope of the least-squares line is

and the intercept is

Here, r is the correlation between y and x, s_y is the standard deviation of y, and s_x is the standard deviation of x. Notice that if the slope is 0, so is the correlation, and vice versa. We discuss this relationship more later in the chapter.

The predicted value of y for a given value x* of x is the point on the least-squares line . This is an unbiased estimator of the mean response μ_y when x = x*. The residualresidual is

The residuals e_i correspond to the linear regression model deviations ϵ_i. The e_i sum to 0, and the ϵ_i come from a population with mean 0. Because we do not observe the ϵ_i, we use the residuals to check the model assumptions of the ϵ_i.

Recall that the least-squares line is the line that minimizes the sum of the squares of the residuals. The least-squares regression line also always passes through the point (). These are helpful facts to remember when considering the fit of this line to a data set.

The remaining parameter to be estimated is σ, which measures the variation of y about the population regression line. Because this parameter is the standard deviation of the model deviations, it should come as no surprise that we use the residuals to estimate it. As usual, we work first with the variance and take the square root to obtain the standard deviation.

For simple linear regression, the estimate of σ² is the average squared residual

562

We average by dividing the sum by n − 2 in order to make s² an unbiased estimate of σ² (the sample variance of n observations uses the divisor n − 1 for this same reason). The quantity n − 2 is called the degrees of freedom for s². The estimate of the model standard deviation σmodel standard deviation σ is given by

We now use statistical software to calculate the regression for predicting BMI from physical activity for Example 10.1. In entering the data, we chose the names PA for the explanatory variable and BMI for the response. It is good practice to use names, rather than just x and y, to remind yourself which variables the output describes.

563

The least-squares regression line is the straight line that is plotted in Figure 10.3. We would report it as

with an estimated model standard deviation of s = 3.655. It says that for each increase of 1000 steps per day, we expect the average BMI to be 0.655 smaller. The large estimated model standard deviation, however, suggests there is a lot of variability about these means.

564

Note that the number of digits provided varies with the software used, and we have rounded the values to three decimal places. It is important to avoid cluttering up your report of the results of a statistical analysis with many digits that are not relevant. Software often reports many more digits than are meaningful or useful.

The outputs contain other information that we will ignore for now. Computer outputs often give more information than we want or need. This is done to reduce user frustration when a software package does not print out the particular statistics wanted for an analysis. The experienced user of statistical software learns to ignore the parts of the output that are not needed for the current analysis.

Because the means μ_y lie on the line μ_y = β₀ + β₁x, they are all determined by β₀ and β₁. Thus, once we have estimates of β₀ and β₁, the linear relationship determines the estimates of μ_y for all values of x. Linear regression allows us to do inference not only for subpopulations for which we have data but also for those corresponding to x’s not present in the data. These x-values can be both within and outside the range of observed x’s. However, extreme caution must be taken when performing inference for an x-value outside the range of the observed x’s because there is no assurance that the same linear relationship between μ_y and x holds.

565

Checking model assumptions

Now that we have fitted a line, we can further check the conditions that the simple linear regression model imposes on this fit. These checks are very important to do and often overlooked. There is no point in trying to do statistical inference if the data do not, at least approximately, meet the conditions that are the foundation for the inference. Misleading or incorrect conclusions can result.

These conditions concern the population, but we can observe only our sample. Thus, in doing inference, we act as if the sample is an SRS from the population. When the data are collected through some sort of random sampling, this assumption is often easy to justify. In other settings, this assumption requires more thought, and the justification is often debatable. For example, in Example 10.1 (page 558), the sample was a collection of volunteers and the researchers argued they could be considered an SRS from the population of college students at that university.

The remaining model conditions can be checked through a visual examination of the residuals. The first condition is that there is a linear relationship in the population, and the second is that the standard deviation of the responses about the population line is the same for all values of the explanatory variable. It is common to plot the residuals both against the case number (especially if this reflects the order in which the observations were collected) and against the explanatory variable. These residual plots are preferred to scatterplots of y versus x because they better magnify patterns.

Figure 10.5 is a plot of the residuals versus physical activity with a smooth-function fit. The scatterplot smoothers are another helpful tool to detect patterns. This smooth function suggests that the average residual is greater than 0 at both low and high physical activity levels. This could mean that a curved relationship between BMI and physical activity would better fit the data. It also could just be the result of chance variation. Notice that there is a large positive residual near each end of the physical activity range that is likely pulling up each end of the smoothed curve. Because the effect does not appear large, we attribute this pattern to chance variation. We do, however, investigate this further in Exercise 11.25 (page 636).

566

To check the assumption of a common standard deviation, we look at the spread of the residuals across the range of x. In Figure 10.5, the spread is roughly uniform across the range of PA, suggesting that this assumption is reasonable. There also do not appear to be any outliers or influential observations.

The final condition is that the response varies Normally about the population regression line. That is, the model deviations vary Normally about 0. Figure 10.6 is a Normal quantile plot of the residuals. Because the plot looks fairly straight, we are confident that we do not have a serious violation of the assumption that the model deviations are Normally distributed.

567

In summary, the data of Example 10.1 give no reason to doubt the simple linear regression model. We can comfortably proceed to inference about parameters, or functions of parameters, such as

If these assumption checks were to raise doubts, it is best to consult an expert, as a more sophisticated regression model is likely needed. There is, however, one relatively simple remedy that may be worth investigation. This is described at the end of this section.

Confidence intervals and significance tests

Chapter 7 presented confidence intervals and significance tests for means and differences in means. In each case, inference rested on the standard errors of estimates and on t distributions. Inference in simple linear regression is similar in principle. For example, the confidence intervals have the form

estimate ± t*SE_estimate

where t* is a critical point of a t distribution. The formulas for the estimate and standard error, however, are more complicated.

As a consequence of the model assumptions about the deviations ϵ_i, the sampling distributions of b₀ and b₁ are Normally distributed with means β₀ and β₁ and standard deviations that are multiples of σ, the model parameter that describes the variability about the true regression line. In fact, even if the ϵ_i are not Normally distributed, a general form of the central limit theorem tells us that the distributions of b₀ and b₁ will be approximately Normal.

Because we do not know σ, we use the estimated model standard deviation s, which measures the variability of the data about the least-squares line. When we do this, we move from the Normal distribution to t distributions with degrees of freedom n − 2, the degrees of freedom of s. We give formulas for the standard errors and in Section 10.2. For now, we concentrate on the basic ideas and let the computer do the computations.

568

Formulas for confidence intervals and significance tests for the intercept β₀ are exactly the same, replacing b₁ and by b₀ and its standard error . Although computer outputs often include a test of H₀: β₀ = 0, this information usually has little practical value. From the equation for the population regression line, μ_y = β₀ + β₁x, we see that β₀ is the mean response corresponding to x = 0. In many practical situations, this subpopulation does not exist or is not interesting.

On the other hand, the test of H₀: β₁ = 0 is quite useful. When we substitute β₁ = 0 in the model, the x term drops out and we are left with

μ_y = β₀

This equation says that the mean of y does not vary with x. In other words, all the y’s come from a single population with mean β₀, which we would estimate by . The hypothesis H₀: β₁ = 0 therefore says that there is no straight-line relationship between y and x and that linear regression of y on x is of no value for predicting y.

We have found a statistically significant linear relationship between physical activity and BMI. The estimated slope is more than 4 standard deviations away from zero. Because this is highly unlikely to happen if the true slope is zero, we have strong evidence for our claim.

Note, however, that this is not the same as concluding that we have found a strong linear relationship between the response and explanatory variables in this example. We saw in Figure 10.3 that there is a lot of scatter about the regression line. A very small P-value for the significance test for a zero slope does not necessarily imply that we have found a strong relationship.

A confidence interval provides additional information about the linear relationship. For most statistical software, these intervals are optional output and must be requested. We can also construct them by hand from the default output.

Note that the intercept in this example is not of practical interest. It estimates average BMI when the activity level is 0, a value that isn’t realistic. For this reason, we do not compute a confidence interval for β₀ or discuss the significance test available in the software.

570

Confidence intervals for mean response

Besides performing inference about the slope (and sometimes the intercept) in a linear regression, we may want to use the estimated regression line to make predictions about the response y at certain values of x. We may be interested in the mean response for different subpopulations or in the response of future observations at different values of x. In either case, we would want an estimate and associated margin of error.

For any specific value of x, say x*, the mean of the response y in this subpopulation is given by

To estimate this mean from the sample, we substitute the estimates b₀ and b₁ for β₀ and β₁:

A confidence interval for μ_y adds to this estimate a margin of error based on the standard error . The formula for the standard error is given in Section 10.2.

Many computer programs calculate confidence intervals for the mean response corresponding to each of the x-values in the data. Some can calculate an interval for any value x* of the explanatory variable. We will use a plot to illustrate these intervals.

571

Some software will do these calculations directly if you input a value for the explanatory variable. Other software will calculate the intervals for each value of x in the data set. Creating a new data set with an additional observation with x equal to the value of interest and y missing will often work.

If we sampled many women who averaged 9000 steps per day, we would expect their average BMI to be between 23.0 and 24.4 kg/m². Note that many of the observations in Figure 10.7 lie outside the confidence bands. These confidence intervals do not tell us what BMI to expect for a single observation at a particular average steps per day. We need a different kind of interval, a prediction interval, for this purpose.

572

Prediction intervals

In the last example, we predicted the average BMI for x* = 9000 steps per day. Suppose that we now want to predict an observation of BMI for a woman averaging 9000 steps per day. The predicted response y for an individual case with a specific value x* of the explanatory variable x is

This is the same as the expression for . That is, the fitted line is used both to estimate the mean response when x = x* and to predict a single future response. We use the two notations and to remind ourselves of these two distinct uses.

This means our best guess for the BMI of this woman averaging 9000 steps per day is what we obtained using the regression equation, 23.7 kg/m². A useful prediction, however, also needs a margin of error (or interval) to indicate its precision. The interval used to predict a future observation is called a prediction intervalprediction interval. Although the response y that is being predicted is a random variable, the interpretation of a prediction interval is similar to that for a confidence interval.

Consider doing the following many times:

Then, 95% of the prediction intervals will contain the value of y for the additional observation. In other words, the probability that this method produces an interval that contains the value of a future observation is 0.95.

The form of the prediction interval is very similar to that of the confidence interval for the mean response. The difference is that the standard error used in the prediction interval includes both the variability due to the fact that the least-squares line is not exactly equal to the true regression line and the variability of the future response variable y around the subpopulation mean. The formula for appears in Section 10.2.

Again, we use a graph to illustrate the results.

The comparison of Figure 10.7 and 10.8 reminds us that the interval for a single future observation must be larger than an interval for the mean of its subpopulation.

Although a larger sample would better estimate the population regression line, it would not reduce the degree of scatter about the line. This means that prediction intervals for BMI, given activity level, will always be wide. This example clearly demonstrates that a very small P-value for the significance test for a zero slope does not necessarily imply that we have found a strong predictive relationship.

Transforming variables

We started our analysis of Example 10.1 with a scatterplot to check whether the relationship between BMI and physical activity could be summarized with a straight line. We followed the least-squares fit with a residual plot (Figure 10.5) and a Normal quantile plot (Figure 10.6) to check Normality, constant standard deviation, and any remaining patterns in the data. A check of model assumptions should always be done prior to inference.

When there is a violation, it is best to consult an expert. However, there are times when a transformation of one or both variables will remedy the situation. In Chapter 2, we discussed the use of the log transformation to describe a curved relationship between x and y. Here is an example where the log transformation has more of an impact on other model assumptions.

The researchers defined entrepreneurs to be those who were self-employed or who were the owner/director of an incorporated business. For each of these individuals, they recorded the education level and income. The education level was defined as the years of completed schooling prior to starting the business. The income level was the average annual total earnings since starting the business.

575

A common remedy for a strongly skewed variable is to consider transforming the variable prior to fitting the model. In this example, the researchers considered the natural logarithm of income (LOGINC).

576

A complete check of the residuals is still needed (see Exercise 10.11), but it appears that transforming y results in a data set that satisfies the linear regression model. Not only is the relationship more linear, but the distribution of the observations about the regression line is more Normal. This is not always the end result of a transformation. In other cases, transforming a variable may help linearity and harm the Normality and constant variance assumptions. Always check the residuals before proceeding with inference.

The long-range goals of the researchers who conducted this study include developing intervention programs (exercise and increasing calcium intake have been shown to be effective) for young women that will increase their TBBMD.