# How to Deal with Multicollinearity?

I use cross-sectional macroeconomics variables with OLS. I found that my data suffers from multicolinearity and i am looking for solutions. I read about first differences of the variables and i tried to do it. However, using first differences the values of the coefficients are totally different and they do not make any sense for interpretating them.

How can I interpret the variables when i take first differences? Are there any solution without needing to take first differences?

Thanks

• There are many ways to handle multicollinearity without having to sacrifice interpretability (as you correctly stated in your post). What is your outcome variable of interest? What independent variables are highly correlated in your regression model? – Matt Reichenbach Apr 10 '15 at 17:01
• My research is on the relationship between debt and growth rate. As dependent variable I use average growth rate of per capita gdp in logs and as independent many others as ratio to real gdp. For instance d/y where d is real debt and y real gdp.I also use one dummy and one interaction term. i know that they increase the multicollinerity but when i remove them from the model it still suffers from multicollinearity. I found high VIF and condition indeces in all of them except from one. – Antonis Apr 10 '15 at 17:38
• List all the variables you are using. Have you transformed the variables you are using yourself? – Plissken Apr 11 '15 at 9:11
• Hi, thank you for yor answer. My dependent variable is : GDP per capita (y_pc) defined as the ln average. The independent variables are: (d/y)=real debt to real GDP, (t/y)= real trade to real GDP, (y_pc0)=initial GDP per capita, (p)=population growth defined with the same way as (y_pc), (i/y)= real investment to real GDP, (g/y)=real govermnment spending to real GDP, (dummy_sec)=south europe countries, (interaction term)=south europe countries. I think the best solution is ridge regression. Do you know how can I apply it on my model with STATA. Note that i am begginer user of stata. Thanks – Antonis Apr 11 '15 at 10:40

I don't understand what you mean by "first differences of the variables". But as far as I know, a very common way to deal with multicollinearity is Penalized regression, like Ridge regression and Lasso;

Take ridge regression as an example to see how it works. First start with the linear regression model $$\boldsymbol y = \mathbf{X}\boldsymbol \beta + \boldsymbol \epsilon$$ where $\boldsymbol \epsilon \ \sim \ N(\mathbf{0}, \sigma^2 \mathbf{I})$, and $\mathbf{X}$ is of full-rank.

The OLS approach is to minimize the residual sum squares (RSS).And the OLS solution is $\hat{\boldsymbol \beta}_{OLS} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X'y}$, and $\mathrm{Var}(\hat{\boldsymbol \beta}) = \sigma^2(\mathbf{X}'\mathbf{X})^{-1}$.

When multicolinearity exists, what happens is $(\mathbf{X}'\mathbf{X})$ will approach to singular and noninvertible. There exists infinite possible solutions that minimize the RSS (There are theories about generalized inverse that helps explain this).

What ridge regression does is that instead of minimizing RSS, it minimizes the following $$RSS + \lambda \sum_{i=1}^p \beta_i^2 \quad (or \quad RSS + \lambda \|\boldsymbol \beta \|^2)$$

The ridge solution is then $$\hat{\boldsymbol \beta}_{Ridge} = (\mathbf{X}'\mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X'y}$$ It can be shown that $(\mathbf{X}'\mathbf{X} + \lambda \mathbf{I})$ is always invertible. When the design matrix is orthonormal, we have $$\hat{\boldsymbol \beta}_{Ridge} =\frac{1}{1 + \lambda} \hat{\boldsymbol \beta}_{OLS}$$

The following simple example shows how ridge regression works when there is collinearity.

> library(MASS)
> set.seed(123)
> n <- 100
> x1 <- rnorm(n, mean=10)
> x2 <- rnorm(n, x1, sd = 0.01)
> y <- 2 + x1 + x2 + rnorm(n)
> lm(y ~ x1 + x2)

Call:
lm(formula = y ~ x1 + x2)

Coefficients:
(Intercept)           x1           x2
3.467       -1.514        3.381

> lm.ridge(y ~ x1 + x2, lambda = 1)
x1        x2
3.5693330 0.9143488 0.9420563


If we increase the correlation of x1 and x2 (change "sd = 0.001"), the effect of collinearity is more clear.

> set.seed(123)
> n <- 100
> x1 <- rnorm(n, mean=10)
> x2 <- rnorm(n, x1, sd = 0.001)
> y <- 2 + x1 + x2 + rnorm(n)
> lm(y ~ x1 + x2)

Call:
lm(formula = y ~ x1 + x2)

Coefficients:
(Intercept)           x1           x2
3.467      -22.944       24.811

> lm.ridge(y ~ x1 + x2, lambda = 1)
x1        x2
3.5703739 0.9267930 0.9295144


Note that the sum of $\hat{\beta}_{OLS}$ would always be close to 2. (if you simulate x2 by setting the mean being equal to a*x1, say a = 10, you will see some interesting yet similar phenomenon.)

Some notes:

1. What ridge regression does is that it shrinks the estimates towards zero but cannot exactly be zero unless $\lambda$ goes to infinity. And if $\lambda$ approaches to 0, the ridge solution converges to OLS solution;
2. It's essentially doing bias-variance trade-off, largely reducing variance of estimates by introducing some bias, leading to smaller mean square error. And this is also one of the main underlying rationals of penalized regression.
3. It cannot do variable selection in the sense that all variables will have non-zero coefficients. Though some can be very small, but are not exactly zero;
4. Lasso regression can do variable selection with L1 penalty.

Another way to deal with multicollinearity is Principle component regression.It basically regresses dependent variable on the principle components of the independent variables obtained by PCA.

• "It basically regresses dependent variable on the principle components of the independent variables obtained by PCA." + These independent variables are now uncorrelated. Very low eigenvalues also indicate high degrees of multicollinearity in the original data. – MaHo Apr 11 '15 at 6:12
• I see thank you very much about the detailed explanation. It seems that I have to use the ridge regression. Do you know any example in STATA in order to study it? note that i am begginer user. Thanks – Antonis Apr 11 '15 at 10:43
• @Antonis, sorry I am not a STATA user. But you can use R (lm.ridge in MASS package) or SAS (PROC REG with "ridge" option) for ridge regression. – Aaron Zeng Apr 12 '15 at 18:15