Suppose two independent variables in the linear regression initially have very high correlation of 0.95. This introduces severe multicollinearity into the model (as indicated by very high variance inflation factors). Can one take natural logarithm of each of them (this decreases correlation between them to 0.75), and use them in the same regression? VIFs do not indicate multicollinearity issues then. Is it a reasonable approach?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ If the variables have a correlation of 0.95 almost all the information given by one of them is included in the information given by the other. Why include both in the regression? $\endgroup$– AghilaCommented Jul 26, 2013 at 7:36
-
$\begingroup$ I used to think like that (that high correlation between variables implies that they carry the same information) until now. By construction and economic meaning, these two variables should be highly correlated. But there is variability of X2 for a given level of X1. $\endgroup$– user28479Commented Jul 26, 2013 at 13:02
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
Sometimes variables are just correlated. It's not necessarily bad, it's just the way that it is.
If you're doing regression, controlling for one variable, that's because you want to control for it. If you do a log transformation, you'll alter the meaning of the variables, you'll alter the distributions of the residuals, and you might introduce non-linear effects. If that all makes sense, go ahead and log them.
In addition, if log transformation reduces the correlation that much, they must have slightly strange distributions. Again, that might be OK, but it might also be something you'll want to look into.
answered Jul 26, 2013 at 1:01
-
1$\begingroup$ +1 But there's a more benign explanation for the reduction in correlation with the logs: please see stats.stackexchange.com/questions/41734/…. No "strangeness" is needed in the distributions, only a small difference in standard deviations. $\endgroup$– whuber ♦Commented Jul 26, 2013 at 1:12
-
$\begingroup$ Thanks for your answers. I take logs of those variables to improve their distribution characteristics (both are highly positively skewed). They both seem to have positive impact on dependent variable in single regressions. When entered both into the same regression, one of them changes sign to minus. Can it be some kind of omitted variable bias? Since when we control for first variable, second one changes sign to negative? And once again, there is non-zero covariance between them. $\endgroup$ Commented Jul 26, 2013 at 1:27
$\begingroup$
$\endgroup$
2
Your results are not entirely surprising. Many variables that represent time series are highly correlated by nature (everything grows over time). Such variables are not only multicollinear to each other. But, they also render regression models heteroskedastic and resulting residuals are autocorrelated (not random). All those issues result into really flawed regression models that can't properly determine the statistical significance of any independent variables used in such a model.
When you transformed the variables to logs, the correlation decreased quite a bit. That's a good thing in most instances. Although taking the log may not entirely rid your regression models of the other potential problems I have mentioned.
You may also want to try converting your independent variables to a % change from one period to another instead of taking the log or just keeping the nominal value as in your original data. When doing so, you should typically see the correlation between the two independent variables drop even a lot more than what you observed when taking logs. Lower correlation between independent variables is good.
-
$\begingroup$ Thanks for your answer! However, my variables do not represent time series. I use cross-sectional data. Therefore, I have no serial correlation in residuals. And also cannot transform my variables into percentage changes. But you are absolutely right about heteroskedasticity (even though this has more to do with large skewness of independent variables distributions, and not collinearity between them) - I use White correction to obtain heteroskedasticity-consistent t-stats. $\endgroup$ Commented Jul 26, 2013 at 4:08
-
$\begingroup$ user28479, I gather given your particular set of circumstances I would run two models. The first one would use your original variables but take out one (the one that is 0.95 correlated to the other, but that has the weakest correlation vs Y). The second model would use your log transformation of those two independent variables. Next, I would look at how those two models compare vs a Hold Out sample... and just see which one works best this way. Both models would be acceptable way to eliminate multicollinearity issues. $\endgroup$– SympaCommented Jul 26, 2013 at 17:11