1
$\begingroup$

One of the assumptions of standard OLS regression is that the regressors are not correlated. But what is the level of correlation at which the assumption is violated? So for instance, if I have three regressors with the following correlations, do they violate the assumption? If so, what can I do to mitigate the effects of collinearity?

+------+------+------+------+
|      | Var1 | Var2 | Var3 |
+------+------+------+------+
| Var1 | 1.0  |      |      |
| Var2 | 0.4  | 1.0  |      |
| Var3 | 0.3  | 0.7  | 1.0  |
+------+------+------+------+
$\endgroup$
  • 2
    $\begingroup$ Welcome to CV! Note, regressors may be correlated. The assumption is more properly stated as little or no multicollinearity. To help you further, what software are you using? R? $\endgroup$ – Thomas Bilach May 19 at 14:29
  • $\begingroup$ Hi! I'm using Stata. Does the low correlation between Var1 and Var3 then imply that there is no multicollinearity despite the high correlation between Var2 and Var3? $\endgroup$ – user770884 May 19 at 15:16
  • $\begingroup$ Pairwise correlations are not reliable indicators of collinearity in multiple regression but they are useful for knowing the appropriate sign (pos/neg) of that relationship. Wrong-signed variables are a useful diagnostic for the presence of collinearity. Partial correlation matrices are useful with VIFs or collinearity indexes provided by many software packages being among the best diagnostic tools. Rules of thumb are available in the literature supporting both of these tools. $\endgroup$ – user332577 May 19 at 16:03
  • $\begingroup$ @user332577 You should make this an answer. $\endgroup$ – Thomas Bilach May 19 at 16:18
  • $\begingroup$ Thanks for the clarification. In that case I'll use a VIF since it seems to be more useful than a simple correlation matrix for testing collinearity. $\endgroup$ – user770884 May 19 at 17:08
4
$\begingroup$

One of the assumptions of standard OLS regression is that the regressors are not correlated

Very very wrong! That's not an assumption of a regression at all. The regressors are almost always correlated unless constructed in a very specific way.

You don't want a perfect multicollinarity, which means that they are Pearson correlated 100%. This is undesirable, usually. If it's not 100% correlation, then it the threshold depends on your objectives. There's no context free threshold upon which you should raise an alarm.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Pairwise correlations are not reliable indicators of collinearity in multiple regression but they are useful for knowing the appropriate sign (pos/neg) of that relationship. Wrong-signed variables are a useful diagnostic for the presence of collinearity. Partial correlation matrices are also useful but VIFs and collinearity indexes provided by many software packages are among the best diagnostic tools. Rules of thumb are available in the literature supporting both of these tools. These include VIFs in the range of around 6 and below.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I have done simulations of linear regression with various levels of multicollinearity and to my great surprise, it seemed to have very little effect. I had a professor tell me that don't even worry about it unless you have > .8 or maybe even > .9 correlation. That turned out to be pretty true in general in my simulations. If I had > .9 I might pick one of the predictors and ignore the other, depending on the situation.

But like some other people have said, VIF's are the best way to look at things. This is because the problem isn't actually correlation, it's linear combinations of the predictors. That is because in the math of regression we have to invert a matrix and you can't do that when there are linear combinations of predictors, if you remember your linear algebra.

If you do have a lot of variables that have high VIF's and you just can't fathom whittling down the predictors, you have some options.

You can do PCA of which is ubiquitous online, so I won't say it here. The other option is QR decomposition.

I learned about it here first: https://mc-stan.org/users/documentation/case-studies/qr_regression.html

That is written in Stan which has a crazy huge learning curve. I little softer is brms, which is a high level interface for Stan.

https://www.rdocumentation.org/packages/brms/versions/2.12.0/topics/brmsformula

Make the model formula and set decomp=TRUE. Then run the formula the command brm https://www.rdocumentation.org/packages/brms/versions/2.12.0/topics/brm

But only try it if you have large numbers of predictors with high VIF's and you can't fathom reducing your predictors.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You were surprised by how little effect there was on prediction or inference? $\endgroup$ – Dave May 20 at 1:02
  • $\begingroup$ That is a good question, I should have been clearer. It appeared that prediction was unaffected, but when correlation >.99, one predictor will become a positive large number, and the other a negative large number. So inference might be off in the presence of high correlation. I don't know about the standard errors, but I will guess that those two will be off as well. $\endgroup$ – Kenney May 20 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.