For high dimensional data, does it make sense to do feature selection before running elastic net?

Question

I have a dataset with $n = 800$ observations and $p = 2000$ features. I'm running elastic net for binary classification.

My question is:

1. Does it make sense to do some feature selection to reduce the number of features to e.g. $p = 100$, before running elastic model? I understand that elastic net and lasso can do 'automatic' feature selection. But I have a pretty high dimensional dataset.

2. If yes, any advice about feature selection methods before running regularized regression models like lasso and elastic net?

Answer 1

Regularization with LASSO, ridge, or elastic net is suitable for even $n<<p$ scenarios, so from the information provided in the question alone, there is no apparent reason to use another method beforehand to pre-select variables.$^\dagger$

However, if you are concerned that there are a lot of non-influential variables, you might consider using LASSO twice,$\ddagger$ re-estimating a subset of the parameters after an initial selection of which are non-zero, which is not an uncommon approach.

In fact, using just elastic net or double LASSO might be a better approach than first selecting variables using some other method, because this may cause the coefficients of the selected variables to be positively biased. Although it focuses more on inferential statistics than classification, you may want to read this related answer, specifically the section:

[A]ll predictors in a model and their posited causal relationship between a single exposure of interest and a single outcome of interest should be specified apriori. Throwing in and excluding covariates based on their relationship between a set of main findings is actually inducing a special case of 'Munchausen's statistical grid' (Martin, 1984).

---

$^\dagger$ Zou & Hastie (2005): Regularization and variable selection via the elastic net
$^\ddagger$ Meinshausen (2006): Relaxed Lasso

Answer 2

Yes, it does make sense. Although, as you stated, these methods do perform feature selection by themselves, reducing the number of predictors is advantageous (reduces effect of the curse of dimensionality), this makes it easier for your classifier to do it's job. For ex. you could remove all features which have very low variance (after standardization), or retain only the 100 variables with highest variance.

Answer 3

Lasso is inherently trying to do feature selection; but it's not the silver bullet. So, yes, it makes sense. There are other techniques by the way, and , additionally, you are not restricted to selection. You could exploit dimensionality reduction techniques such as PCA, LDA, and NMF.