1
$\begingroup$

I am trying to perform a multi linear regression model:

$$y_i = β_0 + β_1x_{i1} + β_2x_{i2} +... + β_px_{ip} + ε_i$$

where $$x_{i1}, x_{i2}, ..., x_{ip}$$ are highly correlated with each other (VIFs can be as low as 5 and high as 10).

I am just wondering if there exists a procedure with the following properties:

1) reduces the collinearity of the variables (e.g. VIFs should be lower than 5 after the procedure)

2) the variables after the procedure should maintain the original meanings/interpretations.. (so PCA and FA are out).

3) not dropping any of the variables. I should have all p original varaibles.. (So LASSO and RIDGE are out)

$\endgroup$
4
  • $\begingroup$ Why do you want to reduce the correlation? $\endgroup$
    – Jonas
    Aug 2, 2018 at 19:08
  • $\begingroup$ @Jonas I am trying to perform "orthogonal transformation" as explained in business.wvu.edu/files/d/d2fd65ec-6fe4-44b5-b3bd-9b84449f3d7e/… to remove the dependency among variables and decompose the variance of the dependent variable. Unfortunately, the similarity, as measured by correlation coefficient, between the original and transformed variable tends to be low if the original variable is highly correlated with other variables. $\endgroup$
    – JungleDiff
    Aug 2, 2018 at 19:11
  • $\begingroup$ Ridge regression estimates don't drop variables... $\endgroup$
    – HStamper
    Aug 2, 2018 at 23:51
  • $\begingroup$ @EricMittman you are right... do you have any suggestions? $\endgroup$
    – JungleDiff
    Aug 2, 2018 at 23:52

1 Answer 1

1
$\begingroup$

Lasso and Ridge regression do not drop the variables but rather it reduces the significance of highly correlated variable by reducing its coefficients.

Also, I need a clarification as to why should you not drop the highly correlated independent variables? As multi-colinearity occurs whenever two Independent variables are highly correlated, in other words, when both the Independent variables are giving you the same information. Having said that, why is it not okay to remove one of those two correlated Independent variable?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.