I have a regression with 6 independent variables and one of them is highly negatively correlated with one of them. (-0.81 correlation) I was wondering if the regression is still valid or if it's biased? Are the variables still orthogonal? The goal of the regression is inference and all variables are important.
$\begingroup$
$\endgroup$
6
-
3$\begingroup$ What is the goal of your regression, prediction or inference? And how interested are you in the doing inference on the two correlated variables? $\endgroup$– DaveCommented Jul 28, 2020 at 18:47
-
$\begingroup$ You could try to use Ridge and Lasso to see which one of these variables are retained in the model with increasing regularity. You can also try bootstrapping/cross validation to assess the stability of the model. $\endgroup$– AtakanCommented Jul 28, 2020 at 19:04
-
1$\begingroup$ (1) The variables never were orthogonal. (2) Correlation does not introduce bias in linear regression. (3) A single high correlation coefficient is rarely of concern of itself. Whether there may be a multicollinearity problem is a matter of analyzing the full six-dimensional distribution of the independent variables. $\endgroup$– whuber ♦Commented Jul 28, 2020 at 19:10
-
$\begingroup$ Try to model one of the two predictors separately and then together. Do you see any substantial difference or inconsistency in their coefficients? If you do Bayesian regression with correlated variables, you may still end up with a stable coefficient, but one of the two may represent the effect conditional on the other variable, so be careful with the inference. statmodeling.stat.columbia.edu/2019/07/07/… $\endgroup$– user289381Commented Jul 28, 2020 at 19:13
-
5$\begingroup$ Does this answer your question? How to deal with high correlation among predictors in multiple regression? $\endgroup$– DurdenCommented Jul 30, 2020 at 2:40
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
Correlation among regressors is perfectly fine, as long as it isn't perfect correlation (i.e., perfect multicollinearity). A correlation of -.81 is no reason for concern. The variance-inflation factor in this case would be :$$VIF = \frac{1}{1-(-0.81)^{2}} \approx 2.9$$ so clearly well below the usual rule-of-thumb cut-off value of 10.
Also, since you've mentioned it: regressors do not need to be orthogonal in linear regression for inference to be valid. You can make them orthogonal (using the Frisch-Waugh theorem), but you don't have to.
