Using elastic-net only for feature selection [duplicate]

Question

Could we use regularization model instead of feature selection methods and then use the machine learning models to analyze data? My problem is classification and there is more than 1000 features in data. The data is mixture of numerical and categorical variables. I would like to use regularization method such as elastic-net for feature selection and then use machine learning model such as random forest to analyze selected features, make prediction and find a feature importance. But I'm not sure this is the correct way!

Answer 1

This is not a good approach, as it is essentially leaking information from the test data.

The resulting coefficients will be overestimated, because no regularization is applied anymore, and yet the data has been used to decide which ones to include. Measures of predictive performance will also be biased, as the same data has been used (in part) to train the model and to evaluate it.

If you want to preselect variables to include in the model, try methods that do not use information about the outcome, such as sparse PCA (on only the features), or create a cross-validation loop that includes the selection procedure, as explained here.

Answer 2

An issue not emphasized in the existing answers is that the elastic net and random forest might not consider the same variables to be important, and then your feature selection step would be denying important features that the random forest could use quite effectively.

For instance, consider this type of image that I have posted on here a few times, such as here.

In this example, the two features are, jointly, outstanding predictors of the color. However, unless you explicitly model the interaction in your penalized logistic regression that does the feature selection, these two features will not be selected. However, a flexible random forest would be able to detect and use this important interaction.

"I'll just include the interaction in my elastic net regression," you say? You have $1000$ variables, so you have almost $1000(1000-1)/2\approx 500,000$ interactions of just two variables, even setting aside the possibilities of three-way (or more) interactions and nonlinear transformations (e.g., squaring) of the variables that a flexible random forest can use.

Answer 3

As you are willing to use a random forest despite the difficulty in making heuristic sense of such models, I suggest that you go directly to a well-designed random forest instead of trying to do (highly problematic) preliminary feature selection. That was already suggested in another answer.

As your predictors include categorical variables, you at least would have to address the problems of standardizing them for proper penalization in the preliminary elastic net that you propose. A well-designed random forest can handle both categorical and continuous predictors directly, take advantage of potential unsuspected interactions, and will minimize overfitting if you let it learn slowly.

At this point you have 3 answers from different perspectives. Why not see for yourself which approach is more reliable by repeating each entire modeling process on resampled data?

This might be done by an optimism bootstrap. Perform a complete modeling process, including feature selection or whatever else is involved, on each of many bootstrap samples. Evaluate each model's performance both on its associated bootstrap sample and on the full data set.

Typically the performance on the associated bootstrap sample will be better than the performance on the full data set. That's an estimate of the "optimism" of the modeling process due to overfitting, which you average over all the models.

This is similar to the approach recommended in a link from the answer by Frans. It evaluates the reliability of your modeling process, which is the best that you can do unless you have the tens of thousands of cases needed for reliable train-test data splits.

What you will almost certainly find is that features identified as "important" will vary greatly among the multiple models on the bootstrap samples, although model performance might nevertheless be OK. Doing the suggested bootstrap validation on each of the approaches you have considered should help you learn for yourself the limitations and dangers of feature selection.

Answer 4

Yes, this is a known and valid approach. In sklearn, this is known as "model-based" selection, and you can generalize it to other models as well, like use a random forest for selection. See https://scikit-learn.org/stable/auto_examples/feature_selection/plot_select_from_model_diabetes.html#sphx-glr-auto-examples-feature-selection-plot-select-from-model-diabetes-py for example.

NB. it is only a valid approach if you learn the selection on the training set, see comment below.

I don't know what your goal is, but you can also immediately use a random forest classifier. This algorithm has a built-in mechanism for selecting features. In terms of predictive performance, I would guess it's going to be more effective to not do this in two steps.