# Coefficients change signs

I have a dataset where there is a high degree of multicollinearity, with all variables correlating positively with each other and the dependent variable. However, on some of the models I run I get a couple of significant negative coefficients. Basically there are two coefficients that depending on what variables I include in the model, I can manipulate their signs.

My understanding is that if the variance-covariance matrix only contain positive values, then all coefficients should also be positive. Is this correct?

• No, this is not true. Try something like the following: Generate a predictor that is standard normal, calling them $X_1$. Generate a second predictor $X_2$ such that $X_2 = 0.99 X_1 + \sqrt{1-0.99^2} Z$ where $Z$ is also standard normal. Now, make your model $Y = X_1 + 0.1 \cdot X_2 + \epsilon$. Everything is pairwise positively correlated, but you're likely to get a negative coefficient estimate on $X_1$ or $X_2$, particularly for small sample sizes like, say, 20. – cardinal Jul 7 '12 at 13:11
• Are you after something like Regression coefficients that flip sign after including other predictors? – chl Jul 7 '12 at 13:12
• A graphical illustration of cardinal's idea can also be found in Why ANOVA/Regression results change when controlling for another variable (near the end of the answer). – caracal Jul 7 '12 at 14:04
• @caracal: (+2, here and there) Another variant uses ellipsoids and so you can see how the major axis changes directions when incribing one inside the other. This captures the idea that the sign of a coefficient is related to the correlation of the predictor with the residuals obtained from regressing out all other predictors. – cardinal Jul 7 '12 at 14:16

Let's generate a small dataset. (Later, you can change this to a huge dataset if you wish, just to confirm that the phenomena shown below do not depend on the size of the dataset.) To get going, let one independent variable $x_1$ be a simple sequence $1,2,\ldots,n$. To obtain another independent variable $x_2$ with strong positive correlation, just perturb the values of $x_1$ up and down a little. Here, I alternately subtract and add $1$. It helps to rescale $x_2$, so let's just halve it. Finally, let's see what happens when we create a dependent variable $y$ that is a perfect linear combination of $x_1$ and $x_2$ (without error) but with one positive and one negative sign.

The following commands in R make examples like this using n data:

n <- 6                  # (Later, try (say) n=10000 to see what happens.)
x1 <- 1:n               # E.g., 1   2 3   4 5   6
x2 <- (x1 + c(-1,1))/2  # E.g., 0 3/2 1 5/2 2 7/2
y <- x1 - x2            # E.g,  1 1/2 2 3/2 3 5/2
data <- cbind(x1,x2,y)


Here's a picture:

First notice the strong, consistent positive correlations among the variables: in each panel, the points trend from lower left to upper right.

Correlations, however, are not regression coefficients. A good way to understand the multiple regression of $y$ on $x_1$ and $x_2$ is first to regress both $y$ and $x_2$ (separately) on $x_1$ (to remove the effects of $x_1$ from both $y$ and $x_2$) and then to regress the $y$ residuals on the $x_2$ residuals: the slope in that univariate regression will be the $x_2$ coefficient in the multivariate regression of $y$ on $x_1$ and $x_2$.

The lower triangle of this scatterplot matrix has been decorated with linear fits (the diagonal lines) and their residuals (the vertical line segments). Take a close look at the left column of plots, depicting the residuals of regressions against $x_1$. Scanning from left to right, notice how each time the upper panel ($x_2$ vs $x_1$) shows a negative residual, the lower panel ($y$ vs $x_1$) shows a positive residual: these residuals are negatively correlated.

That's the key insight: multiple regression peels away relationships that may otherwise be hidden by mutual associations among the independent variables.

For the doubtful, we can confirm the graphical analysis with calculations. First, the covariance matrix (scaled to simplify the presentation):

> cov(data) * 40
x1 x2  y
x1 140 82 58
x2  82 59 23
y   58 23 35


The positive entries confirm the impression of positive correlation in the scatterplot matrix. Now, the multivariate regression:

> summary(lm(y ~ x1+x2))
...
Estimate Std. Error    t value Pr(>|t|)
(Intercept) -7.252e-16  2.571e-16 -2.821e+00   0.0667 .
x1           1.000e+00  1.476e-16  6.776e+15   <2e-16 ***
x2          -1.000e+00  2.273e-16 -4.399e+15   <2e-16 ***


One slope is +1 and the other is -1. Both are significant.

(Of course the slopes are significant: $y$ is a linear function of $x_1$ and $x_2$ with no error. For a more realistic example, just add a little bit of random error to $y$. Provided the error is small, it can change neither the signs of the covariances nor the signs of the regression coefficients, nor can it make them "insignificant.")

• (+1) (Inside joke: Well that's one way to handle a flag...) :) – cardinal Jul 13 '12 at 14:50
• In this example, it seems like multiple regression reveals the 'true' relationship between y and x2 (as y=x1-x2) if I understand correctly. Could a sign-flip of a coefficient be misleading in another instance with two highly collinear covariates? – Blain Waan Feb 25 at 6:17
• @Blain Interesting question. My sense is that if there aren't numerical issues in the solution (which can be checked), then the result will not be misleading. The collinearity causes the standard errors of estimate to be higher than you might expect, but all that will do is require you to use more data to achieve a desirable level of confidence in the coefficient estimates. Another way to put it is if the results are statistically significant, they won't be any more or less misleading than in any other circumstance. – whuber Feb 25 at 15:30
• There is a previous post where it said: "if your variables are positively correlated, then the coefficients will be negatively correlated, which can lead to a wrong sign on one of the coefficients" and that ridge regression can help get the corrected signs. I have also seen that if I choose a $\lambda$ high enough, such sign flips reverse and align with what would be found in the bivariates. I'm confused if we should do that at all. The link to the post I am talking about: stats.stackexchange.com/questions/1580/… – Blain Waan Feb 25 at 17:12
• "Can lead" is merely a possibility. Assuming the model is close to correct, then when the coefficients are precisely estimated and significantly different from zero, it's unlikely they have the wrong signs, regardless of collinearity. The claims in that particular answer are vague, general, and not specifically supported with examples or analysis, so you will need to be cautious about applying them. – whuber Feb 25 at 17:38