I want to perform classification of breast cancer cases by using models like SVM or Random Forest. When I was browsing the web I saw that one could use Lasso for feature selection, and then applied these features as input to my ML models. The question that I have is if its useful to use Lasso as a feature selector for classification tasks. I ask this, because for what I know Lasso is used in Regression models. Could I use Lasso as a feature selector in classification tasks? Thanks
$\begingroup$
$\endgroup$
1
-
$\begingroup$ @Dave is right that it's possible for things to go wrong in several ways. But the only way to know for sure if it's going to work in your tasks or not is to try it out. The way to use lasso in classification tasks is to use it for a "binomial GLM" or "multinomial GLM" (for two class and multi-class problems, respectively). In R, you could use the package "glmnet", using the setting family = "binomial" or "multinomial". $\endgroup$– John MaddenCommented Oct 26, 2023 at 16:27
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
THERE IS NO REASON TO EXPECT THIS TO WORK
LASSO selects features for a linear model. There is no reason to expect these to be the key features once you start allowing for nonlinearities and interactions that are not present in that original model. For instance, it might be that $X_3$ is a terrible predictor of the outcome when you just consider a linear combination of predictors as is done in, say, a logistic regression (to which it sounds like you would be applying the LASSO penalty), yet there is a quadratic relationship between $X_3$ and the outcome that the random forest or support vector classifier is able to detect. By flitering out the unselected $X_3$ variable, you deny the random forest and support vector models the opportunity to determine if $X_3$ is important to the way they model. In my example here, a logistic regression cannot make accurate predictions from either feature and would regard both as being unimportant. However, a particular kernel trick in an SVM would handle that X shape just fine.
Indeed, some of the point of using models like random forests and support vector machines is that they discover such behavior without being explicitly told, so there is a sense in which you are denying these models the opportunity to do what they exist to do.
Finally, LASSO, like other feature selection methods, is often unstable in what it selects. Frank Harrell discusses this in a $2020$ keynote address to Why R? (a recording is here) where he recommends bootstrapping the procedure and weeping over how much variability there is in the features that get selected. He has other posts about feature selection instability if you search through his posts on his profile here. One such post is here.