11
$\begingroup$

Does it ever make sense to check for multicollinearity and perhaps remove highly correlated variables from your dataset prior to running LASSO regression to perform feature selection?

One of the scientists I am working with is highly concerned that by not dealing with multicollinearity before LASSO regression, the LASSO model will perform poorly, though I'm not sure what the general consensus is for this. I was thinking that because LASSO will shrink some coefficients to zero, multicollinearity is remedied. Any thoughts or suggestions?

Improve this question
$\endgroup$
13
  • 3
    $\begingroup$ Will you be using the LASSO to select features and then fit a regression on those features? // What is the purpose of the modeling, pure prediction? $\endgroup$
    – Dave
    Commented Jul 15, 2021 at 15:34
  • 1
    $\begingroup$ Not to stir the pot, but if variables are highly collinear (and highly predictive), then LASSO will on average tend to select just one and discard the other with no preference. As far as a predictive routine, that seems just fine by reckoning, i.e. given two predictive variables with the same information I don't care which one I use, just that I use one of them. $\endgroup$
    – AdamO
    Commented Jul 15, 2021 at 15:45
  • 1
    $\begingroup$ @Dave Yes, our goal is to use LASSO to perform feature selection and ultimately run a logistic model using the coefficients selected by LASSO (as well as clinical variables we know are important). The model will be used for prediction purposes. $\endgroup$
    – user122514
    Commented Jul 15, 2021 at 15:48
  • 1
    $\begingroup$ So why not run the regularized regression on all of your variables and cross validate to find the hyperparameter giving the best performance? $\endgroup$
    – Dave
    Commented Jul 15, 2021 at 15:49
  • 1
    $\begingroup$ Why does that matter if that's what gives the best predictive performance? // It sounds like she wants to use those 200 features in an OLS regression, using LASSO for feature selection. Is that accurate? $\endgroup$
    – Dave
    Commented Jul 15, 2021 at 16:02

3 Answers 3

Reset to default
1
$\begingroup$

Yes, removing multicollinear predictors before LASSO can be done, and may be a suitable approach depending on what you are trying to accomplish with the model. If you are interested in estimating if there are significant predictors of some response variable(s), then what removing multicollinear predictors will do is lessen the variance inflation of the standard errors of your regression parameters.

LASSO will reduce the absolute size of your regression parameters, but that is not the same thing as the standard errors of those parameters.

Edit

An update to one of the comments under the original post. Hyperparameter tuning using a cross validation scheme should give you relatively optimal results for purely predictive purposes. If it is feasible, do it both ways so that you and your scientific colleague learn more about what empirically works best for your use case.

Improve this answer
$\endgroup$
1
$\begingroup$

Your colleague has a valid concern. While LASSO regression can handle multicollinearity to some extent by shrinking coefficients of correlated predictors, it's still a good practice to check for multicollinearity before running LASSO.

Interpretability: High multicollinearity can make it difficult to interpret the coefficients of the model, as the estimates may become unstable and sensitive to small changes in the model.

Model Performance: Although LASSO can reduce the impact of multicollinearity by selecting a subset of predictors, having highly correlated variables can still affect the model's performance and accuracy.

Feature Selection: LASSO performs feature selection by shrinking some coefficients to zero, but it might not always select the most relevant features if multicollinearity is present.

Improve this answer
$\endgroup$
-4
$\begingroup$

Do the following and you should be fine:

  1. Run a LASSO regression to select variables
  2. Check correlation/VIF on the selected variables (correlated variables lead to misleading estimates of your coefficients because of an unstable hyperplane)
  3. Re-Estimate your p-values, standard errors and so on in a model with only the selected variables

This way you avoid wrong estimates of coefficients due to multicollinear variables, and your p-values and standard errors are no longer biased.

Improve this answer
$\endgroup$
24
  • $\begingroup$ It is not appropriate to do analyses of only the selected variables as these disrespect the shrinkage built in to lasso. Also, collineaerity will ruin the feature selection so needs to be dealt with before lasso. $\endgroup$
    – Frank Harrell
    Commented Jun 17 at 11:44
  • $\begingroup$ There is a difference between "in theory" and "in practice". I am yet to see a single example where this does approach does not work. $\endgroup$
    – Esben Eickhardt
    Commented Jun 24 at 7:52
  • $\begingroup$ I’d wager that calibration curves are not an integral part of your analysis then. You lose calibration when you do this. Calibration (absolute predictive accuracy; calibration-in-the-small) is all-important. $\endgroup$
    – Frank Harrell
    Commented Jun 24 at 11:38
  • 2
    $\begingroup$ I image that your idea of “work” is much different from the mainstream. You will not be able to demonstrate absolute predictive accuracy. Plus in ideal situations with no collinearities, lasso does not do a good job in selecting the right variables: fharrell.com/talk/stratos19 $\endgroup$
    – Frank Harrell
    Commented Jun 27 at 12:59
  • 1
    $\begingroup$ Unwillingness to even attempt to unbiased smooth calibration curves is almost always a symptom of being afraid that a method is not as good as promised. The only reason to avoid calibration curves is to avoid embarrassment. $\endgroup$
    – Frank Harrell
    Commented Oct 21 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.