A couple things:
...following two methods can be used for feature selection prior to model development
Those are actually part of the model development and should be cross validated. What most people do is look at the correlations, select only the most correlated with the output, and then move on to do cross validation etc. That's wrong for two reasons.
- Correlation measures strength of a linear relationship. If the effect of the variable is non linear, your correlation may not pick up on this. Here is a concrete example. When I worked for a marketing company, most of the customers we dealt with were in their late 30s to early 40s. They spent the most money, and people younger and older spent less because young people typically didn't have as much money or interest in our products. So the effect of age kind of looked like a concave function. If you simulate something like
x = rnorm(1000); y = -0.2*x^2 + rnorm(1000, 0, 0.5) (here x has a concave relationship to y) the correlation is low even though x can explain 75% of the variation observed in y. If you removed features based on their correlation, surely you would not select this very important feature.
- Had you had different training data, you might have picked different features. So when you fit models in the cross validation step, you need to repeat the selection of features based on the correlations. Same thing with the lasso. In every cross validation refit, you need to fit the lasso, select the features, then refit a model with the selected features.
Which of the above two methods is preferred?
I don't think either are very good to be honest. Correlation is myopic for the reasons I've described. Lasso is better, but there is no reason to think the features it selects are the "best" features, nor is there a reason to think that the features it selects would be selected had you had different data. Here is a code example to demonstrate that
library(tidyverse)
library(glmnet)
S = 0
N = 1000
p = 100
mu = rep(0, p)
betas = rnorm(p, 2, 2)*rbinom(p, 1, 0.10)
while(max(abs(S))<0.9){
S = rethinking::rlkjcorr(1, p)
}
do_glmnet<-function(){
X = MASS::mvrnorm(N, mu, S)
y = X %*% betas + rnorm(N, 0, 2.5)
cvmodel = cv.glmnet(X,y, alpha = 1)
model = glmnet(X, y, alpha = 1)
coef(model, cvmodel$lambda.1se) %>%
as.matrix() %>%
t() %>%
as_tibble()
}
results = map_df(1:100, ~do_glmnet())
results %>%
summarise_all(~mean(abs(.)>0) )
In that example, I generate data from a sparse linear model. Some variables are selected every time (those are the variables with real effects) but you can see that some variables with 0 effect are sometimes selected and sometimes not selected.
The absolute best way to select features is to use your knowledge about the data generating process to determine what is important and what is not. If you can't do that, use lasso to trade off variance for a but of bias, but don't select out features. Just keep the entire fit in the model. I saw Trevor Hastie speak at a zoom talk the other day and he showed us an example of which LASSO with all features performed better than selecting features with LASSO and then refitting the full model. I can't say that is the case for every problem, but it was pretty compelling evidence. Let me see if I can find a link to the talk.
That being said, I'm open to seeing numerical experiments that show that selection via glmnet does better than just putting everything into glmnet and not selecting. That just hasn't been the story I've seen.
