# What do changes in regression coefficients indicate about correlations among predictors?

How do you tell if there is a strong, weak, or no correlation between two predictors if you are only given the regression coefficients from two models. One model contains one predictor and the other model contains both predictors.

• (1) You can't tell a thing about correlations among predictors (independent variables) based on coefficients in a regression. Are you perhaps asking about correlations between a predictor and a response variable? (2) Are these the actual coefficients or the standardized coefficients? The former tell you absolutely nothing about the correlation (after all, the coefficients can be arbitrarily changed--even negated--simply by rescaling one or both variables), while the standardized coefficients are related to correlation. – whuber Mar 11 '14 at 17:49
• @whuber: if the coefficient of a covariable changes considerably when entering a second covariable, this is a hint that the two covariables could highly be correlated. – Michael M Mar 11 '14 at 18:00
• @MichaelMayer, so if the values of the two regression coefficients are only different by 0.005, then does that mean they are not correlated, or are weakly correlated? – Ben Mar 11 '14 at 18:03
• @Michael You assume a lot in drawing that conclusion, such as the two models involve exactly the same set of cases. Even then the change in one coefficient can come about because the new explanatory variable is highly correlated with the dependent variable: the amount of correlation between the two explanatory variables can be arbitrarily small. – whuber Mar 11 '14 at 18:20
• You can't tell a thing with just the $\hat\beta$'s alone. Often you'll have subject-matter knowledge which can help you to know whether to expect a correlation. It is possible to have the beta's be significant even in the presence of high correlation between predictors in a dataset, if the sample size is big enough. Google the "variance inflation factor" and convince yourself that it becomes unimportant as sample size gets big. – generic_user Mar 11 '14 at 18:21

Sometimes an example shows something. In this case, my example shows that @whuber is right: You can't tell much.

Here's a model with uncorrelated predictors

set.seed(123)  #set random seed
x1 <- rnorm(100)
x2 <- rnorm(100)
y <- x1 + x2 + rnorm(100)

m1 <- lm(y~x1)
m2 <- lm(y~x1 + x2)

coef(m1)
coef(m2) #coef for x1 changes from 0.81 to 0.86
cor(x1,x2) #0.04


Here is one with highly correlated predictors

x3 <- x1 + rnorm(100,0,1)
y2 <- x1 + x3 + rnorm(100)
m3 <- lm(y~x1 + x3)
coef(m1)
coef(m3) #coef for x1 changes from .81 to .82
cor(x1,x3) #.64


In one case, the change is .05, in the other .01.

• Thx Peter. Here, the changes in the coefficient is very small. Can you contradict my comment criticized by @whuber by another example? – Michael M Mar 11 '14 at 21:15
• @MichaelMayer You said they could be highly correlated and they certainly could. But need they be? That is, could you have a large change in a coefficient by adding a variable that is not highly correlated with the other independent variable? Sure, just so long as there is an interaction. This in fact happens; one case: Trying to predict whether a person has HIV. Individual variable 1 = sex (male, female, other) IV 2 = sexuality (straight, gay/lesbian, bi, other) Adding that second variable changes the first coefficient a LOT. – Peter Flom Mar 11 '14 at 21:25
• i agree if you add the interaction term. But if you don't add it as in the OP? – Michael M Mar 11 '14 at 21:33
• I am not sure. Maybe @whuber will add more – Peter Flom Mar 11 '14 at 22:34

This question can be resolved with a single picture, shown below, which immediately shows the answer is in the negative: you cannot conclude anything about the strength of the correlation between two predictors by comparing the coefficients in two models. The text of this answer explains the picture and the reasoning behind it.

### Background

Many questions about relationships among variables in multiple regression (using ordinary or generalized least squares) end up being matters of plane Euclidean geometry. The reasons for this are

1. The variables (independent and dependent) in multiple regression, when written as columns in a table of $$n$$ data rows, can be viewed as vectors in $$n$$ dimensional Euclidean space.

2. The least squares fit is the orthogonal projection of the dependent variable on the subspace spanned by the independent variables.

3. Multiple regression can be conducted as a sequence of "linear matches". At each step there are vectors $$x_1, \ldots, x_k$$ and $$y$$ representing $$k+1$$ independent variables and a vector $$z$$ representing the dependent variable. Relative to $$y$$, we may decompose any vector $$w$$ uniquely into the sum

$$w = w_{\cdot y} + (w - w_{\cdot y})$$

where $$w_{\cdot y}$$ is parallel to $$y$$ and the residual $$w - w_{\cdot y}$$ is orthogonal to $$y$$. For the next step, all the other vectors $$\{x_i\}$$ and $$z$$ are replaced by their residuals with respect to $$y$$ and one of the new $$x_i$$ is chosen to be the next $$y$$. This process is illustrated below.

Notice that when a "constant" is included in the regression, step (3) amounts to recentering all the variables: the constant $$y$$ is represented as the $$n$$-vector $$(1,1,\ldots,1)$$, whence $$w_{\cdot y} = (\bar{w}, \bar{w}, \ldots, \bar{w})$$, and $$(w - w_{\cdot y})$$ subtracts the mean $$\bar{w}$$ from each component of $$w$$. Thus "matching" is a natural generalization of recentering.

Due to the recursion afforded by (3), almost anything we would like to learn about multiple regression can be discovered by considering just three vectors $$x, y,$$ and $$z$$. Moreover, when all is done, $$z$$ will have been projected into the plane generated by $$\{x,y\}$$, so when we are concerned only about the coefficients (and not the residuals), we can ignore the third dimension (determined by the direction of $$z$$ orthogonal to the $$\{x,y\}$$ plane).

(One can go further and standardize the vectors $$x$$ and $$y$$. This means they can be assumed to have unit length; standardization merely amounts to choosing suitable units of measurement for the underlying coordinates. However, re-standardizing the vectors after each matching step can change the apparent correlations among them, so I forgo that simplification here.)

Consequently, we can analyze and intuitively understand this question using simple planar figures.

### Analysis

In the illustration, the betas are least squares coefficients:

• $$\beta_x$$ is the coefficient of $$x$$ when matching $$x$$ alone to $$z$$; in other words, $$\beta_x x$$ is the projection of $$z$$ onto the line determined by $$x$$.

• $$\beta_{(x,y)}$$ is the coefficient of $$x$$ in the regression of $$z$$ against $$x$$ and $$y$$. Equivalently, it is the coefficient of $$x_{\cdot y}$$ when projecting $$z_{\dot y}$$ in the second step of the matching process: $$y$$ has been taken out of $$z$$ and $$x$$.

Moreover, the correlation coefficient $$\rho$$ between the predictors $$x$$ and $$y$$ is the cosine of the angle $$\theta$$ between them.

### Solution

The question, translated into geometry, asks

What can be said about $$\rho = \cos(\theta)$$ (whether it is near $$\pm 1$$, near $$0$$ or exactly $$0$$) if we are only given $$\beta_x$$ and $$\beta_{(x,y)}$$?

In the illustration $$\beta_x$$ is near $$2/3$$ and $$\beta_{(x,y)}$$ is about $$-3/2$$, while $$\theta$$ is fairly small, corresponding to small positive correlation. This already shows that large changes in the coefficients of $$x$$ can occur even when $$x$$ and $$y$$ are not strongly correlated.

When $$\rho\ne 0$$, the vectors $$x$$ and $$x_{\cdot y}$$ are not parallel, whence they generate the entire $$\{x,y\}$$ plane. Accordingly, arbitrary values of $$\beta_x$$ and $$\beta_{(x,y)}$$ determine unique values of $$z$$ for which the coefficients of $$x$$ in the two models are $$\beta_x$$ and $$\beta_{(x,y)}$$. $$z$$ is constructed by drawing the line perpendicular to $$x$$ at the point $$\beta_x x$$ and the line perpendicular to $$x_{\cdot y}$$ at the point $$\beta_{(x,y)}x_{\cdot y}$$; their intersection is $$z$$.

This proves that little can be concluded about $$\theta$$ (or, equivalently, $$\rho$$) regardless of what the two coefficients happen to be. (As we ought to expect, those coefficients say much more about the dependent variable $$z$$ than they do about the independent variables.)

When $$\rho=0$$, $$\beta_x = \beta_{(x,y)}$$. However, as the preceding paragraph has shown, the converse--equality of the two betas--does not imply $$\rho=0$$.

Therefore, the only conclusion that can be supported is that if the coefficient of $$x$$ changes in the two regressions, then $$x$$ and $$y$$ must be correlated. (The textbooks usually state the contrpositive: when $$x$$ and $$y$$ are not correlated, adding $$y$$ to the model will not change the coefficient of $$x$$.) However, the degree of correlation can be any nonzero value between $$-1$$ and $$1$$ inclusive, so we cannot draw any conclusion about the strength of the correlation.