2
$\begingroup$

I was reading about stability selection applied to LASSO. My understanding is that stability selection (Meinshausen & Buhlman, 2010) helps in finding stable variables, with error control provided by an upper bound, but it doesn't provide an optimized regularisation parameter, so I'd say that it doesn't really do model selection. I see how using only the subset of stable variables identified in this way to train further models might be wrong. So my question is how feature selection performed with stability can be integrated in a predictive framework?

Imagine I have a classification problem with binary outcome. I am interested in assessing the prediction performance of LASSO and identify stable predictors and their sign (association with the outcome).

The only way I see is to perform a LASSO using, for instance, a nested k-fold cross-validation and evaluate model performance.

Then perform stability selection as a separate analysis to identify significant predictors. I expect that the subset of variables selected in each loop of the nested cv will be, at least, slightly different from the stable variables identified by stability selection. My question is, is it ok to evaluate model performance with a metod and predictors importance with another method? Is there a way to extract coefficient signs from stability selection procedure (I'm using stabsel in R).

Improve this question
$\endgroup$
2
  • $\begingroup$ are you talking about Meinshausen & Buhlman stability selection? $\endgroup$
    – rep_ho
    Apr 16, 2021 at 9:52
  • $\begingroup$ Yep, that's exactly what I was talking about. I've edited the post. $\endgroup$
    – Pecto
    Apr 16, 2021 at 9:55

1 Answer 1

Reset to default
1
$\begingroup$

I am thinking about stability selection more like a statistical procedure to give you a set of significant variables in a high dimensional setting, where traditional linear regression would fail. That it uses LASSO in the paper, which is usually used for machine learning purposes is kind of a red herring.

Now, as you correctly noticed, it cannot be easily used for model selection, but you can use it for feature selection, where the selected features by stability selection will be likely better than features selected by LASSO alone. In this case, you will first select variables using stabsel, and then fit a model that does not have to be LASSO, since you don't want to do another feature selection on these variables alone. You don't need to only take the selected variables, since the stabsel selection is quite conservative. Just be sure you do it inside your CV loop to avoid double-dipping.

Another possibility is to run and validate your LASSO the same way as you do. But then as a supplementary analysis also run stability selection to tell you which variables are significant. Since prediction and identifying statistically significant variables are separate goals, you can have separate analyses for that.

ps: stability selection works for any feature selection method, not just LASSO, lasso is just very convenient since it can produce the whole regularization path in one go

Improve this answer
$\endgroup$
5
  • $\begingroup$ That is really clear, thanks for that! About the first option you provide, I would totally do it but I'm concerned about using the same dataset to perform a feature selection first and as a separate step model selection and evaluation with the feature subset previously selected. I've always thought that these two steps must be integrated in some way (e.g. SVM with RFE in a double nested k fold for parameter tuning and backward selection). Am I right? $\endgroup$
    – Pecto
    Apr 16, 2021 at 10:34
  • $\begingroup$ you can do anything you want, as far as you test it on a dataset that was not used for feature selection, i.e., make feature selection in the training set separately for each cv fold, and not in the whole dataset. You can use double nested CV if you also want to choose for example what's the best number of features to select $\endgroup$
    – rep_ho
    Apr 16, 2021 at 10:37
  • $\begingroup$ I see what you mean, thanks! Just another question, do you think I can apply stabilty selection to a Random Forest? Instead of using non-zero coefficients (as in LASSO), using kind of top k ranked features? I ask, because I've never seen that case in literature but just lasso and boosted gaussian regressions $\endgroup$
    – Pecto
    Apr 16, 2021 at 10:46
  • $\begingroup$ I don't know, but people don't do it for random forests and probably there is a reason for that. Bootstrapping random forests IMO won't help, since well, they are already random, and there are other RF specific methods that might work better or at least be easier to work with. Mathematically, there is no condition on what feature selection method to use to get valid set of selected features (I think) $\endgroup$
    – rep_ho
    Apr 16, 2021 at 10:52
  • $\begingroup$ Great! This was very helpful! $\endgroup$
    – Pecto
    Apr 16, 2021 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.