Correlation measures the direction and strength of the straight-line (linear) relationship between two quantitative variables. If a scatterplot shows a linear relationship, we would like to summarize this overall pattern by drawing a line on the scatterplot. A regression line summarizes the relationship between two variables, but only in a specific setting: when one of the variables helps explain or predict the other. That is, regression describes a relationship between an explanatory variable and a response variable.

81

Suppose we want to use this relationship between GDP per capita and net assets per capita to predict the net assets per capita for a country that has a GDP per capita of $50,000. To predict the net assets per capita (in thousands of dollars), first locate 50 on the x axis. Then go “up and over” as in Figure 2.9 to find the GDP per capita that corresponds to . We predict that a country with a GDP per capita of $50,000 will have net assets per capita of about $200,000.

The least-squares regression line

Different people might draw different lines by eye on a scatterplot. We need a way to draw a regression line that doesn't depend on our guess as to where the line should be. We will use the line to predict from , so the prediction errors we make are errors in , the vertical direction in the scatterplot. If we predict net assets per capita of 177 and the actual net assets per capita are 170, our prediction error is

82

The error is .

No line will pass exactly through all the points in the scatterplot. We want the vertical distances of the points from the line to be as small as possible.

There are several ways to make the collection of vertical distances “as small as possible.” The most common is the least-squares method.

One reason for the popularity of the least-squares regression line is that the problem of finding the line has a simple solution. We can give the recipe for the least-squares line in terms of the means and standard deviations of the two variables and their correlation.

83

We write (read “y hat”) in the equation of the regression line to emphasize that the line gives a predicted response for any . Because of the scatter of points about the line, the predicted response will usually not be exactly the same as the actually observed response . In practice, you don't need to calculate the means, standard deviations, and correlation first. Statistical software or your calculator will give the slope and intercept of the least-squares line from keyed-in values of the variables and . You can then concentrate on understanding and using the regression line. Be warned—different software packages and calculators label the slope and intercept differently in their output, so remember that the slope is the value that multiplies in the equation.

The slope of a regression line is almost always important for interpreting the data. The slope is the rate of change, the amount of change in when increases by 1. The slope in this example says that each additional $1000 of GDP per capita is associated with an additional $4500 in net assets per capita.

The intercept of the regression line is the value of when . Although we need the value of the intercept to draw the line, it is statistically meaningful only when can actually take values close to zero. In our example, occurs when a country has zero GDP. Such a situation would be very unusual, and we would not include it within the framework of our analysis.

84

To plot the line on the scatterplot, you can use the equation to find for two values of , one near each end of the range of in the data. Plot each above its , and draw the line through the two points. As a check, it is a good idea to compute for a third value of and verify that this point is on your line.

How many digits should we keep in reporting the results of statistical calculations? The answer depends on how the results will be used. For example, if we are giving a description of the equation, then rounding the coefficients and reporting the equation as would be fine. If we will use the equation to calculate predicted values, we should keep a few more digits and then round the resulting calculation as we did in Example 2.12.

Facts about least-squares regression

Regression as a way to describe the relationship between a response variable and an explanatory variable is one of the most common statistical methods, and least squares is the most common technique for fitting a regression line to data. Here are some facts about least-squares regression lines.

Fact 1. There is a close connection between correlation and the slope of the least-squares line. The slope is

This equation says that along the regression line, a change of one standard deviation in corresponds to a change of standard deviations in . When the variables are perfectly correlated , the change in the predicted response is the same (in standard deviation units) as the change in . Otherwise, because , the change in is less than the change in . As the correlation grows less strong, the prediction moves less in response to changes in .

Fact 2. The least-squares regression line always passes through the point on the graph of against . So the least-squares regression line of on is the line with slope that passes through the point . We can describe regression entirely in terms of the basic descriptive measures , and .

Fact 3. The distinction between explanatory and response variables is essential in regression. Least-squares regression looks at the distances of the data points from the line only in the direction. If we reverse the roles of the two variables, we get a different least-squares regression line.

87

Interpretation of r²

The square of the correlation describes the strength of a straight-line relationship. Here is the basic idea. Think about trying to predict a new value of . With no other information than our sample of values of , a reasonable choice is .

Now consider how your prediction would change if you had an explanatory variable. If we use the regression equation for the prediction, we would use . This prediction takes into account the value of the explanatory variable .

Let's compare our two choices for predicting . With the explanatory variable , we use ; without this information, we use , the sample of the response variable. How can we compare these two choices? When we use to predict, our prediction error is . If, instead, we use , our prediction error is . The use of in our prediction changes our prediction error from is to . The difference is . Our comparison uses the sums of squares of these differences and . The ratio of these two quantities is the square of the correlation:

The numerator represents the variation in that is explained by , and the denominator represents the total variation in .

88

When you report a regression, give as a measure of how successful the regression was in explaining the response. The software outputs in Figure 2.11 include , either in decimal form or as a percent. When you see a correlation (often listed as R or Multiple R in outputs), square it to get a better feel for the strength of the association.

Residuals

A regression line is a mathematical model for the overall pattern of a linear relationship between an explanatory variable and a response variable. Deviations from the overall pattern are also important. In the regression setting, we see deviations by looking at the scatter of the data points about the regression line. The vertical distances from the points to the least-squares regression line are as small as possible in the sense that they have the smallest possible sum of squares. Because they represent “leftover” variation in the response after fitting the regression line, these distances are called residuals.

89

California spends $5.73 million less on education than the least-squares regression line predicts. On the scatterplot, the residual for California is shown as a dashed vertical line between the actual spending and the least-squares line.

90

As usual, when we perform statistical calculations, we prefer to display the results graphically. We can do this for the residuals.

91

If the regression line captures the overall relationship between and , the residuals should have no systematic pattern. The residual plot will look something like the pattern in Figure 2.15(a). That plot shows a scatter of points about the fitted line, with no unusual individual observations or systematic change as increases. Here are some things to look for when you examine a residual plot:

92

The distribution of the residuals

When we compute the residuals, we are creating a new quantitative variable for our data set. Each case has a value for this variable. It is natural to ask about the distribution of this variable. We already know that the mean is zero. We can use the methods we learned in Chapter 1 to examine other characteristics of the distribution. We will see in Chapter 10 that a question of interest with respect to residuals is whether or not they are approximately Normal. Recall that we used Normal quantile plots to address this issue.

Take a look at the plot of the data with the least-squares line in Figure 2.2 (page 67). Note that you can see the same four points in this plot. If we eliminated these states from our data set, the remaining residuals would be approximately Normal. On the other hand, there is nothing wrong with the data for these four states. A complete analysis of the data should include a statement that they are somewhat extreme relative to the distribution of the other states.

Influential observations

In the scatterplot of spending on education versus population in Figure 2.12 (page 87) California, Texas, Florida, and New York have somewhat higher values for both variables than the other 46 states. This could be of concern if these cases distort the least-squares regression line. A case that has a big effect on a numerical summary is called influential.

93

Let's think about a situation in which California would be influential on the least-squares regression line. California's spending on education is $110.1 million. This case is close to both least-squares regression lines in Figure 2.17. Suppose California's spending was much less than $110.1 million. Would this case then become influential?

94

California, Texas, Florida, and New York are somewhat unusual and might be considered outliers. However, these cases are not influential with respect to the least-squares regression line.

Influential cases may have small residuals because they pull the regression line toward themselves. That is, you can't always rely on residuals to point out influential observations. Influential observations can change the interpretation of data. For a linear regression, we compute a slope, an intercept, and a correlation. An individual observation can be influential for one of more of these quantities.

The best way to grasp the important idea of influence is to use an interactive animation that allows you to move points on a scatterplot and observe how correlation and regression respond. The Correlation and Regression applet on the text website allows you to do this. Exercises 2.73 and 2.74 later in the chapter guide the use of this applet.