To move from using multiple regression for data analysis to inference in the multiple regression setting, we need to make some assumptions about our data. These assumptions are summarized in the form of a statistical model. As with all the models that we have studied, we do not require that the model be exactly correct. We only require that it be approximately true and that the data do not severely violate the assumptions.

Recall that the simple linear regression model assumes that the mean of the response variable depends on the explanatory variable according to a linear equation

For any fixed value of , the response varies Normally around this mean and has a standard deviation that is the same for all values of .

In the multiple regression setting, the response variable depends on not one but explanatory variables, which we denote by . The mean response is a linear function of these explanatory variables:

549

Similar to simple linear regression, this expression is the population regression equation, and the observed ’s vary about their means given by this equation.

Just as we did in simple linear regression, we can also think of this model in terms of subpopulations of responses. The only difference is that each subpopula-tion now corresponds to a particular set of values for all the explanatory variables . The observed ’s in each subpopulation are still assumed to vary Normally with a mean given by the population regression equation and standard deviation that is the same in all subpopulations.

Multiple linear regression model

To form the multiple regression model, we combine the population regression equation with assumptions about the form of the variation of the observations about their mean. We again think of the model in the form

The FIT part of the model consists of the subpopulation mean . The RESIDUAL part represents the variation of the response around its subpopulation mean. That is, the model is

The symbol represents the deviation of an individual observation from its sub-population mean. We assume that these deviations are Normally distributed with mean 0 and an unknown standard deviation that does not depend on the values of the variables.

The assumption that the subpopulation means are related to the regression coefficients by the equation

implies that we can estimate all subpopulation means from estimates of the ’s. To the extent that this equation is accurate, we have a useful tool for describing how the mean of varies with any collection of ’s.

550

Estimating the parameters of the model

To estimate the mean U.S. box office revenue in Example 11.12, we must estimate the coefficients , , , and . Inference requires that we also estimate the variability of the responses about their means, represented in the model by the standard deviation .

In any multiple regression model, the parameters to be estimated from the data are , and . We estimate these parameters by applying least-squares multiple regression as described in Section 11.1. That is, we view the coefficients in the multiple regression equation

as estimates of the population parameters . The observed variability of the responses about this fitted model is measured by the variance

and the regression standard error

551

In the model, the parameters and measure the variability of the responses about the population regression equation. It is natural to estimate by and by .

Inference about the regression coefficients

Confidence intervals and significance tests for each of the regression coefficients have the same form as in simple linear regression. The standard errors of the ’s have more complicated formulas, but all are again multiples of . Statistical software does the calculations.

552

Be very careful in your interpretation of the t tests and confidence intervals for individual regression coefficients. In simple linear regression, the model says that . The null hypothesis says that regression on is of no value for predicting the response , or alternatively, that there is no straight-line relationship between and . The corresponding hypothesis for the multiple regression model of Example 11.13 says that is of no value for predicting y, given that and are also in the model. That’s a very important difference.

The output in Figure 11.11 shows, for example, that the -value for opening-weekend theater count is . We can conclude that the number of theaters does not help predict U.S. box office revenue, given that budget and opening-weekend revenue are available to use for prediction. This does not mean that the opening-weekend theater count cannot help predict U.S. box office revenue. In Exercise 11.33 (page 550) you showed there was a strong positive relationship between the number of theaters and total U.S. revenue, especially when the number of theaters was greater than 2500.

554

The conclusions of inference about any one explanatory variable in multiple regression depend on what other explanatory variables are also in the model. This is a basic principle for understanding multiple regression. The tests in Example 11.13 show that the opening-weekend theater count does not significantly aid prediction of the U.S. box office revenue if the budget and opening-weekend revenue are also in the model. On the other hand, opening-weekend revenue is highly significant even when the budget and opening-weekend theater count are also in the model.

The interpretation of a confidence interval for an individual coefficient also depends on the other variables in the model, but in this case only if they remain constant. For example, the 95% confidence interval for Opening implies that, given the number of theaters and the budget do not change, a $1 million increase in the opening-weekend revenue results in an expected increase in total U.S. box office revenue somewhere between $1.42 and $2.80 million. While it makes sense for the budget to remain fixed, it may not make sense to keep the number of theaters fixed. The number of theaters and opening-weekend revenue are positively correlated, and it may be very unreasonable to assume that opening revenue can increase this much without the number of theaters also increasing.

Inference about prediction

Inference about the regression coefficients looks much the same in simple and multiple regression, but there are important differences in interpretation. Inference about prediction also looks much the same, and in this case the interpretation is also the same. We may wish to give a confidence interval for the mean response for some specific set of values of the explanatory variables. Or we may want a prediction interval for an individual response for the same set of values.

The distinction between predicting a mean and individual response is exactly as in simple regression. The prediction interval is again wider because it must allow for the variation of individual responses about the mean. In most software, the commands for prediction inference are the same for multiple and simple regression. The details of the arithmetic performed by the software are, of course, more complicated for multiple regression, but this does not affect interpretation of the output.

555

What about changes in the model, which we saw can greatly influence inference about the regression coefficients? It is often the case that different models give similar predictions. We expect, for example, that the predictions of U.S. box office revenue from budget and opening-weekend revenue will be about the same as predictions based on budget, opening-weekend revenue, and opening-weekend theater count. Because of this, when prediction is the key goal of a multiple regression, it is common to search for a model that predicts well but does not contain unnecessary predictors. Some refer to this as following the KISS principle.¹¹ In Section 11.3, we discuss some procedures that can be used for this type of search.

ANOVA table for multiple regression

The basic ideas of the regression ANOVA table are the same in simple and multiple regression. ANOVA expresses variation in the form of sums of squares. It breaks the total variation into two parts: the sum of squares explained by the regression equation and the sum of squares of the residuals. The ANOVA table has the same form in simple and multiple regression except for the degrees of freedom, which reflect the number of explanatory variables. Here is the ANOVA table for multiple regression:

The brief notation in the table uses, for example, MSE for the residual mean square. This is common notation; the "E’’ stands for "error.’’ Of course, no error has been made. "Error’’ in this context is just a synonym for "residual.’’

556

The degrees of freedom and sums of squares add, just as in simple regression:

The estimate of the variance for our model is again given by the MSE in the ANOVA table. That is, .

The ratio MSR/MSE is again the statistic for the ANOVA test. In simple linear regression, the test from the ANOVA table is equivalent to the two-sided test of the hypothesis that the slope of the regression line is . These two tests are not equivalent in multiple regression.

In the multiple regression setting, the null hypothesis for the test states that all the regression coefficients (with the exception of the intercept) are . One way to write this is

A shorter way to express this hypothesis is

The alternative hypothesis is

The null hypothesis says that none of the explanatory variables helps explain the response, at least when used in the form expressed by the multiple regression equation. The alternative states that at least one of them is linearly related to the response.

This test provides an overall assessment of the model to explain the response. The individual tests assess the importance of a single variable given the presence of the other variables in the model. While looking at the set of individual tests to assess overall model significance may be tempting, it is not recommended because it leads to more frequent incorrect conclusions. The test also better handles situations when there are two or more highly correlated explanatory variables.

As in simple linear regression, large values of give evidence against . When is true, has the distribution. The degrees of freedom for the distribution are those associated with the regression and residual terms in the ANOVA table.

The distributions are a family of distributions with two parameters: the degrees of freedom of the mean square in the numerator and denominator of the statistic. The distributions are another of R. A. Fisher’s contributions to statistics and are called in his honor. Fisher introduced statistics for comparing several means. We meet these useful statistics in Chapters 14 and 15.

The numerator degrees of freedom are always mentioned first. Interchanging the degrees of freedom changes the distribution, so the order is important. Our brief notation will be for the distribution with degrees of freedom in the numerator and in the denominator. The distributions are not symmetric but are right-skewed. The density curve in Figure 11.12 illustrates the shape. Because mean squares cannot be negative, the statistic takes only positive values, and the distribution has no probability below 0. The peak of the density curve is near 1; values much greater than 1 provide evidence against the null hypothesis.

Tables of critical values are awkward, because a separate table is needed for every pair of degrees of freedom and . Table E in the back of the book gives upper critical values of the distributions for and .

557

558

A significant test does not tell us which explanatory variables explain the response. It simply allows us to conclude that at least one of the coefficients is not zero. We may want to refine the model by eliminating some variables that do not appear to be useful (KISS principle). On the other hand, if we fail to reject the null hypothesis, we have found no evidence that any of the coefficients are not zero. In this case, there is little point in attempting to refine the model.

Squared multiple correlation

For simple linear regression, the square of the sample correlation can be written as the ratio of SSR to SST. We interpret as the proportion of variation in explained by linear regression on . A similar statistic is important in multiple regression.

Often, is multiplied by 100 and expressed as a percent. The square root of , called the multiple regression correlation coefficient, is the correlation between the observations and the predicted values . Some software provides a scatterplot of this relationship to help visualize the predictive strength of the model.

The statistic for the multiple regression of U.S. box office revenue on budget, opening-weekend revenue, and opening-weekend theater count is highly significant, . There is strong evidence of a relationship among these three variables and eventual box office revenue. The squared multiple correlation tells us that these variables in this multiple regression model explain about 72% of the variability in box office revenues. The other 28% is represented by the RESIDUAL term in our model and is due to differences among the movies that are not measured by these three variables. For example, these differences might be explained by the movie’s rating, the genre of the movie, and whether the movie is a sequel.

559

Inference for a collection of regression coefficients

We have studied two different types of significance tests for multiple regression. The test examines the hypothesis that the coefficients for all the explanatory variables are zero. On the other hand, we used tests to examine individual coefficients. (For simple linear regression with one explanatory variable, these are two different ways to examine the same question.)

Often, we are interested in an intermediate setting: does a set of explanatory variables contribute to explaining the response, given that another set of explanatory variables is also available? We formulate such questions as follows: start with the multiple regression model that contains all the explanatory variables and test the hypothesis that a set of the coefficients are all zero. When this set involves more than one explanatory variable, we need to consider an test rather than a set of individual parameter tests.

Some software allows you to directly state and test hypotheses of this form. Here is a way to find the statistic by doing two regression runs.

560

The hypothesis test in Example 11.16 asks about the coefficients of Budget and Theaters in a model that also contains Opening as an explanatory variable. If we start with a different model, we may get a different answer. For example, we would not be surprised to find that Budget and Theaters help explain movie revenue in a model with only these two explanatory variables. Individual regression coefficients, their standard errors, and significance tests are meaningful only when interpreted in the context of the other explanatory variables in the model.