5
$\begingroup$

Does it ever make sense to check for multicollinearity and perhaps remove highly correlated variables from your dataset prior to running LASSO regression to perform feature selection?

One of the scientists I am working with is highly concerned that by not dealing with multicollinearity before LASSO regression, the LASSO model will perform poorly, though I'm not sure what the general consensus is for this. I was thinking that because LASSO will shrink some coefficients to zero, multicollinearity is remedied. Any thoughts or suggestions?

$\endgroup$
13
  • 3
    $\begingroup$ Will you be using the LASSO to select features and then fit a regression on those features? // What is the purpose of the modeling, pure prediction? $\endgroup$
    – Dave
    Jul 15, 2021 at 15:34
  • 1
    $\begingroup$ Not to stir the pot, but if variables are highly collinear (and highly predictive), then LASSO will on average tend to select just one and discard the other with no preference. As far as a predictive routine, that seems just fine by reckoning, i.e. given two predictive variables with the same information I don't care which one I use, just that I use one of them. $\endgroup$
    – AdamO
    Jul 15, 2021 at 15:45
  • $\begingroup$ @Dave Yes, our goal is to use LASSO to perform feature selection and ultimately run a logistic model using the coefficients selected by LASSO (as well as clinical variables we know are important). The model will be used for prediction purposes. $\endgroup$
    – user122514
    Jul 15, 2021 at 15:48
  • 1
    $\begingroup$ So why not run the regularized regression on all of your variables and cross validate to find the hyperparameter giving the best performance? $\endgroup$
    – Dave
    Jul 15, 2021 at 15:49
  • $\begingroup$ @Dave, that was my initial thought process and what I presented to my colleague yesterday. However, using cross-validation to select the "best" lambda gave us a model with a little over 200 features. That's when she immediately felt concerned about LASSO keeping too many correlated features. $\endgroup$
    – user122514
    Jul 15, 2021 at 15:57

1 Answer 1

0
$\begingroup$

Yes, removing multicollinear predictors before LASSO can be done, and may be a suitable approach depending on what you are trying to accomplish with the model. If you are interested in estimating if there are significant predictors of some response variable(s), then what removing multicollinear predictors will do is lessen the variance inflation of the standard errors of your regression parameters.

LASSO will reduce the absolute size of your regression parameters, but that is not the same thing as the standard errors of those parameters.

Edit

An update to one of the comments under the original post. Hyperparameter tuning using a cross validation scheme should give you relatively optimal results for purely predictive purposes. If it is feasible, do it both ways so that you and your scientific colleague learn more about what empirically works best for your use case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.