4
$\begingroup$

The very essence of lasso is that it is supposed to select only one of two correlated variables.

However, when I include two highly correlated predictors (they are correlated with each other at level ~0.95), both of them are being selected with similar absolute coefficient value (on standardized predictor), but with different signs. This means their effect on prediction almost cancels out, but the coefficients from model on standardized input are highest of all variables.

Example:

          x         coefs 
 (Intercept)        91.6958266
 Population_2013   -49.2656083
 Population_2014    46.8513210 

where Variable1 and Variable2 are highly correlated. Other correlated and uncorrelated variables are also included in model. I run models on anything between 20 and 20000 variables and effect is similar for these correlated variables.

Is there any solution? Alternatively - how in any other way can I determine which variables affect significantly my prediction?

$\endgroup$
4
  • 2
    $\begingroup$ Use ridge regression or elastic net when you have highly correlated variables. $\endgroup$ – Richard Hardy May 20 '16 at 16:17
  • 2
    $\begingroup$ What is the objective of your analysis? And what do you mean "run models on anything between 20 and 20,000 variables"? I don't agree that "the very essence of lasso is that it is supposed to select only one of two correlated variables". Yes, the lasso is relatively unstable with correlated variables (small changes in $Y$ can change the solution), but that is only a side effect of the $\ell_1$ penalty on the sum of squares, not a desideratum. $\endgroup$ – Andrew M May 20 '16 at 16:40
  • 2
    $\begingroup$ From Richard Hastie's glmnet package vignette: "It is known that the ridge penalty shrinks the coefficients of correlated predictors towards each other while the lasso tends to pick one of them and discard the others." - this is what I mean by essence of lasso w.r.t. correlated variables. I don't understand how ridge regression can help here (actually - it does not change anything). I run models on different business financial indicators with initially large number of automatically generated hypothesis to test, I delete these I cannot attribute meaningful causality to my dependent variable. $\endgroup$ – user2530062 May 21 '16 at 20:36
  • 1
    $\begingroup$ Actually someone else suggested to me it might be lack of penalization - lambda chosen by algorithm to minimize CV error is too low and correlated variables are being allowed into the model. I will investigate it on Monday, I will appreciate other suggestions though. $\endgroup$ – user2530062 May 21 '16 at 20:44
4
$\begingroup$

The answer turned out to be simple: lambda was low so there was no regularization, therefore lasso did not work as expected. Solution was to manually select lambda instead of relying on lambda minimizing CV error.

$\endgroup$
1
  • 3
    $\begingroup$ That'll be the explanation (+1); it's not so clear what problem this is a solution to. To "determine which variables affect significantly my prediction"? I don't think so - as @AndrewM points out LASSO can be quite arbitrary in which of correlated predictors it shrinks away. At least bootstrap & see how consistently the same set of predictors is kept. $\endgroup$ – Scortchi - Reinstate Monica May 23 '16 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.