Often, we have many explanatory variables, and our goal is to use these to explain the variation in the response variable. A model using just a few of the variables often predicts about as well as the model using all the explanatory variables. We may also find that the reciprocal of a variable is a better choice than the variable itself or that including the square of an explanatory variable improves prediction. How can we find a good model? That is the model building issue. A complete discussion would be quite lengthy, so we must be content with illustrating some of the basic ideas with a Case Study.

The analysis starts with a study of the variables involved. Here is a short summary of this work.

567

Price, as we expect, has a right-skewed distribution. The mean (in thousands of dollars) is $158 and the median is $130. There is one high outlier at $830, which we delete as unusual in this location. Remember that a skewed distribution for Price does not itself violate the conditions for multiple regression. The model requires that the residuals from the fitted regression equation be approximately Normal. We have to examine how well this condition is satisfied when we build our regression model.

BedRooms ranges from one to five. The website uses five for all homes with five or more bedrooms. The data contain just one home with one bedroom. Baths includes both full baths (with showers or bathtubs) and half baths (which lack bathing facilities). Typical values are 1, 1.5, 2, and 2.5. Garage has values of 0, 1, 2, and 3. The website uses the value 3 when three or more vehicles can fit in the garage. There are 50 homes that can fit three or more vehicles into their garage (or possibly garages). The data set has begun a process of combining some values of these variables, such as five or more bedrooms and garages that hold three or more vehicles. We continue this process as we build models for predicting Price.

Zip describes location, traditionally the most important explanatory variable for house prices, but Zip is a quite crude description because a single zip code covers a broad area. All of the postal zip codes in this community have 4790 as the first four digits. The fifth digit is coded as the variable Zip. The possible values are 1, 4, 5, 6, and 9. There is only one home with zip code 47901. We first look at the houses in each zip code separately.

SqFt, the number of square feet for the home, is a quantitative variable that we expect to strongly influence Price. We start our analysis by examining the relationship between Price and this explanatory variable. To control for location, we start by examining only the homes in zip code 47904, corresponding to . Most homes for sale in this area are moderately priced.

568

Because the relationship is approximately linear and we expect SqFt to be an important explanatory variable, let’s start by examining the simple linear regression of Price on SqFt.

Models for curved relationships

Figure 11.13 suggests that the relationship between SqFt and Price may be slightly curved. One simple kind of curved relationship is a quadratic function. To model a quadratic function with multiple regression, create a new variable that is the square of the explanatory variable and include it in the regression model. There are now explanatory variables, and . The model is

with the usual conditions on the .

The output in Figure 11.15 is a good example of the need for care in interpreting multiple regression. The individual tests for both SqFt and SqFt2 are not significant. Yet, the overall test for the null hypothesis that both coefficients are zero is significant (, , ). To resolve this apparent contradiction, remember that a test assesses the contribution of a single variable, given that the other variables are present in the model. Once either SqFt or SqFt2 is present, the other contributes very little. This is a consequence of the fact that these two variables are highly correlated. This phenomenon is called collinearity or multicollinearity. In extreme cases, collinearity can cause numerical instabilities, and the results of the regression calculations can become very imprecise. Collinearity can exist between seemingly unrelated variables and can be hard to detect in models with many explanatory variables. Some statistical software packages will calculate a variance inflation factor (VIF) value for each explanatory variable in a model. VIF values greater than 10 are generally considered an indication that severe collinearity exists among the explanatory variables in a model. Exercise 11.80 (page 583) explores the calculation and use of VIF values. In this particular case, we could dispense with either SqFt or SqFt2, but the test tells us that we cannot drop both of them. It is natural to keep SqFt and drop its square, SqFt2.

Multiple regression can fit a polynomial model of any degree:

In general, we include all powers up to the highest power in the model. A relationship that curves first up and then down, for example, might be described by a cubic model with explanatory variables , , and . Other transformations of the explanatory variable, such as the square root and the logarithm, can also be used to model curved relationships.

571

Models with categorical explanatory variables

Although adding the square of SqFt failed to improve our model significantly, Figure 11.13 (page 567) does suggest that the price rises a bit more steeply for larger homes. Perhaps some of these homes have other desirable characteristics that increase the price. Let’s examine another explanatory variable.

"Number of bedrooms’’ is now a categorical variable that places homes in two groups: one/two bedrooms and three/four bedrooms. Software often allows you to simply declare that a variable is categorical. Then the values for the two groups don’t matter. We could use the values 2 and 3 for the two groups. If you work directly with the variable, however, it is better to indicate whether or not the home has three or more bedrooms. We will take the "number of bedrooms’’ categorical variable to be if the home has three or more bedrooms and if it does not. Bed3 is called an indicator variable.

572

We need only variables to code different values because the last value is identified by "all indicator variables are 0.’’

Example 11.21 shows that regression on one indicator variable essentially models the means of the two groups. A regression model for a categorical variable with possible values requires indicator variables and therefore regression coefficients (including the intercept) to model the category means. Indicator variables can also be used in models with other variables to allow for different regression lines for different groups. Exercise 11.76 (page 582) explores the use of an indicator variable in a model with another quantitative variable for this purpose.

Here is an example of a categorical explanatory variable with four possible values.

More elaborate models

We now suspect that a model that uses square feet, number of bedrooms, and number of bathrooms may explain price reasonably well. Before examining such a model, we use an insight based on careful data analysis to improve the treatment of number of baths.

Figure 11.18 reminds us that the data describe only two homes with 2.5 bathrooms. Our model with three indicator variables fits a mean for these two observations. The pattern of Figure 11.18 reveals an interesting feature: adding a half bath to either one or two baths raises the predicted price by a similar, quite substantial amount. If we use this information to construct a model, we can avoid the problem of fitting one parameter to just two houses.

So far we have learned that the price of a home is related to the number of square feet, whether or not there are three or more bedrooms, whether or not there is an additional full bathroom, and whether or not there is an additional half bathroom. Let’s try a model including all these explanatory variables.

Variable selection methods

We have arrived at a reasonably satisfactory model for predicting the asking price of houses. But it is clear that there are many other models we might consider. We have not used the Garage variable, for example. What is more, the explanatory variables can interact with each other. This means that the effect of one explanatory variable depends upon the value of another explanatory variable. We account for this situation in a regression model by including interaction terms.

The simplest way to construct an interaction term is to multiply the two explanatory variables together. Thus, if we wanted to allow the effect of an additional half bath to depend upon whether or not there is an additional full bath, we would create a new explanatory variable by taking the product . Interaction terms that are the product of an indicator variable and another variable in the model can be used to allow for different slopes for different groups. Exercise 11.76 (page 582) explores the use of such an interaction term in a model.

578

Considering interactions increases the number of possible explanatory variables. If we start with the five explanatory variables SqFt, Bed3, B2, Bh, and Garage, there are 10 interactions between pairs of variables. That is, there are now 15 possible explanatory variables in all. From 15 explanatory variables, it is possible to build 32,767 different models for predicting price. We need to automate the process of examining possible models.

Modern regression software offers variable selection methods that examine all possible multiple regression models. The software then presents us a list of the top models based on some selection criteria. Available criteria include , the regression standard error s, AIC, and BIC. These latter three criteria balance fit against model simplicity in different ways and are appropriate for comparing models with different numbers of explanatory variables. should only be used when comparing two models with the same number of explanatory variables.

There are only 37 houses in the data set. A model with too many explanatory variables will fit the accidental wiggles in the prices of these specific houses and may do a poor job of predicting prices for other houses. This is called overfitting.

There is no formula for choosing the "best’’ multiple regression model. We tend to prefer smaller models to larger models because they avoid overfitting. However, too small a model may not fit the data or predict well. Model selection criteria such as the regression standard error or AIC balance these two goals but in different ways. Using one of these is preferred to when trying to determine a best model because does not account for overfitting.

We also want a model that makes intuitive sense. For example, the best single predictor () is SqFtBh, the interaction between square feet and having an extra half bath. In general, we would not use a model that has interaction terms unless the explanatory variables involved in the interaction are also included. The variable selection output does suggest that we include SqFtBh and, therefore, SqFt and Bh.

The negative coefficient for Bh seems odd. We expect an extra half bath to increase price. A plot will help us to understand what this model says about the prices of homes.

We can determine the equations of the two lines from the fitted regression equation. Homes that lack an extra half bath have . When we set in the regression equation

we get

For homes that have an extra half bath, and the regression equation becomes

In Figure 11.24, we graphed this line starting at the price of the least-expensive home with an extra half bath. These two equations tell us that an additional square foot increases asking price by $15.15 for homes without an extra half bath and by $67.96 for homes that do have an extra half bath.

BEYOND THE BASICS: Multiple Logistic Regression

Many studies have yes/no or success/failure response variables. A surgery patient lives or dies; a consumer does or does not purchase a product after viewing an advertisement. Because the response variable in a multiple regression is assumed to have a Normal distribution, this methodology is not suitable for predicting yes/no responses. However, there are models that apply the ideas of regression to response variables with only two possible outcomes.

581

The most common technique is called logistic regression, which we discuss in Chapter 17. The starting point is that if each response is 0 or 1 (for failure or success), then the mean response is the probability of a success. Logistic regression tries to explain in terms of one or more explanatory variables. Details are even more complicated than those for multiple regression, but the fundamental ideas are very much the same and software handles most details. Here is an example.

Interpretation of the coefficients is a little more difficult in multiple logistic regression because of the form of the model. The expression is the odds that the model is sexually dressed. Logistic regression models the "log odds’’ as a linear combination of the explanatory variables. Positive coefficients are associated with a higher probability that the model is dressed sexually. We see that ads in magazines with younger readers, female models, and women’s magazines are more likely to show models dressed sexually.

Similar to the test in multiple regression, there is a chi-square test for multiple logistic regression that tests the null hypothesis that all coefficients of the explanatory variables are zero. The value is , and the degrees of freedom are the number of explanatory variables, four in this case. The -value is reported as . (You can verify that it is less than 0.0005 using Table F.) We conclude that not all the explanatory variables have zero coefficients.

In place of the tests for individual coefficients in multiple regression, chi-square tests, each with one degree of freedom, are used to test whether individual coefficients are zero. For reader age and model gender, , while for the indicator for women’s magazines, . The indicator for men’s magazines is not statistically significant.