2
$\begingroup$

In a linear regression I have two variables that are correlated with rho = 0.8. Given two multiple linear models where the two variables go in mutually exclusive, the estimare for each one is highly statistical significant. If I plug in both variables, the significance almost vanishes for both, likely due to the high correlation among the two variables

Unfortunately, I cannot collect anymore data to tackle the problem. So I thought maybe ridge and/or lasso regression can help: Both methods can be used in case of high collinearity among the regressors and can be used for model selection. As standard errors for ridge and lasso regressions are mostly meaningess, my reasoning is now: look at a ridge or lasso model at "optimal" lambda value and check if one or both of the variables are driven out of the model (estimate (close to) zero) by the shrinkage or close to being driven out (for larger lambda values).

Does that sound like a sane suggestion to support an effect of both variables? I am not really interested in the estimates themselves (maybe their sign) but rather want to know if both of them stand for an independent effect (on their own) in the model when including both.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ It depends on your objectives. If you need estimates of both coefficients as part of understanding a theory, for instance, then it will do you no good to leave out either one of the variables--you would just have to live with the high standard errors. $\endgroup$
    – whuber ♦
    Oct 23, 2017 at 21:39
  • 2
    $\begingroup$ Instead of the two separate t-tests of the two coefficients, look at the F-test of both together. That might well be significant even if both t-tests are not. $\endgroup$
    – kjetil b halvorsen ♦
    Oct 23, 2017 at 21:44
  • $\begingroup$ Good point, whuber. I added a sentence at the very end to elaborate on my objective. $\endgroup$
    – Helix123
    Oct 23, 2017 at 21:46
  • $\begingroup$ I would look at the joint CI for the two coefficients, like this. $\endgroup$
    – dimitriy
    Oct 24, 2017 at 1:48
  • 1
    $\begingroup$ @kjetil b halvorsen yes the F test for joint significance shows joint significance. $\endgroup$
    – Helix123
    Oct 24, 2017 at 18:36

1 Answer 1

Reset to default
1
$\begingroup$

LASSO is not designed to deal with collinearity. Ridge is. I would use ridge regression here.

I would either bootstrap the standard errors or else (and I'd rather do this) ignore the question of statistical significance and look at the size of the estimates and figure out whether they are big enough to consider important.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Oh, I thought all regularisation techniques are suitable in presence of high multicollinearity. How come LASSO is not? It is based on the same reasoning, isn't it? $\endgroup$
    – Helix123
    Oct 24, 2017 at 7:55
  • 3
    $\begingroup$ That's been discussed here e.g. stats.stackexchange.com/questions/241471/… and stats.stackexchange.com/questions/25611/… $\endgroup$
    – Peter Flom
    Oct 24, 2017 at 10:56
  • 1
    $\begingroup$ What I can read from those is, LASSO is not appropriate for exactly collinear variables. However, I only have a high correlation, no linear dependence between the two. From the original paper about LASSO I read that it designed as combining dealing with a high degree of collinearity as in ridge regression and model selection in one step. $\endgroup$
    – Helix123
    Oct 24, 2017 at 11:55
  • 1
    $\begingroup$ You can use elastic net which is a linear combination of lasso and ridge $\endgroup$
    – kjetil b halvorsen ♦
    Oct 24, 2017 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.