Does the feature selection matter for learning algorithm with regularization?

Question

Let's assume we have infinite computing power. When we consider two algorithms,

1. learning algorithm + regularization and
2. feature selection + (learning algorithm + regularization),

Which one would achieve better prediction performance usually?

---

Now, my original post on feature selection vs regularization is duplicate with the post. I'd like to update my question focusing on the point which is still ambiguous.

I read the previous answers and they are not arriving at one conclusion. The answers point to the opinion that it depends on the situation. I'd like to narrow my question to focus on the case which still is not considered.

I summarize some of the views from the answers:

1. This answer mentions that it depends on the learning algorithm. It says that random forest would do better than the NN for selecting relevant features among large numbers of features. So the random forest would need feature selection less than NN.

2. This answer indicates that the data size for training can be the issue. The feature selection might put one more layer of training into the fitting procedure and leads to overfitting because the feature selection is done at the subset of data.

3. This answer mentions that if the application requires refit repeatedly with new data, then, the wrong feature might affect the performance, and feature selection would be helpful.

4. This answer indicates that it depends on the regularization method.

The answers consider various factors that can affect the feature selection performance. But I think there might be several still more factors to consider. In this updated question, I'd like to raise the issue on the signal to noise ratio.

My first thought is this: If the signal to noise ratio of the data set is low, there is more danger of overfitting and I feel that extra step of feature selection might help because it can remove the irrelevant feature catching the noise in a different way. In this setting, removing features might be more important than keeping more features.

However, from the opposite point of view, the extra step of feature selection might lead to more severe overfitting with argument 2.

Both views make sense to me at the moment.

How much is the feature selection(preprocessing) helpful for the learning algorithm with regularization training with the data samples of different levels of signal to noise ratios?

Answer 1

Short answer:

In my opinion you can achieve better results considering (learning algorithm + regularization) than (feature selection + learning algorithm + regularization). Of course, it depends on the regularization technique selected (see long answer below) but the main advantage of using a (learning algorithm + regularization) like lasso is that the process of variable selection is being made with the objective of minimizing your loss function, while many feature selection methods that are not embedded inside the learning algorithm do not take the loss function information into account.

Long answer:

There are many feature selection methods. Usually, these methods are classified into three categories:

• Filter based: We compute a metric and perform feature selection based on this metric. For example, pearson correlation coefficient. The main disadvantage of this approach is that when performing the feature selection we are not taking into account what is the objective of the learning algorithm that we are going to perform. So the feature selection step is independent of the learning algorithm and variables that are significant for the learning algorithm are at risk of being discarded.

• Wrapper-based: Either forward selection, backward elimination or any other alternative based on those. The main idea is always the same, define a subset of variables, solve your problem for such a subset of variables, define a new subset, solve the problem again,... Iterate until you find the optimal subset. In this approach we solve the disadvantage of the filter based methods, because now the variable selection is linked to our learing algorithm objective, but the problem here is that wrapper based methods are computationally really expensive since we need to tune one model for each subset considered. Additionally, those methods are really data dependent, meaning that if you change your training dataset because you add some new observations, you can end up with an entirely different subset of variables.

• Embedded: Finally, embedded methods. These methods use algorithms that have built-in feature selection methods. Here we include many of the regularization techniques such as LASSO or SCAD penalizations. Embedded methods are, as the name says, embedded inside the learning algorithm, so they are capable of performing variable selection and prediction at the same time. This means that the variable selection is performed taking into account the learning algorithm objective. Additionally, those methods are generally continuous processes, meaning that they are more robust than wrapper based methods against changes on the observations of the dataset (your feature selection is more stable). The "disadvantage" of these methods is that they usually include a hyperparameter that controls the level of penalization applied. Eg: if we are using a LASSO penalization in OLS, the objective function would be:

$$ \sum_{i=1}^n(y_i-\beta^tx_i)^2 + \lambda\sum_{j=1}^p\|\beta_j\|_1 $$

where $\lambda$ is controlling the level of penalization applied. Large $\lambda$ values give more weight to the penalization producing solutions that are more sparse. Small $\lambda$ produce less penalized solutions. On the limit, if $\lambda=0$ then we are not penalizing at all. Generally, the value of $\lambda$ is tuned using some criteria such as grid search and cross validation.

This said, it is important to remark that the effect achieved by the regularization process greatly depends on the regularization itself. For example:

• ridge regression penalizes in terms of an $L_2$ norm, and thus yields to solutions that are more robust against colinearity (which is a comon problem when dealing with high dimensional data in which the number of variables is very large). But ridge does not perform any kind of feature selection. For this reason, ridge can be useful when dealing with colinear datasets / medium sized number of variables, but it is not suited for variable selection.

• Lasso penalization works in terms of an $L_1$ norm and thus perfroms automatic variable selection, because it will send to $0$ some of the coefficients of your model. The variables associated to coefficients with value $0$ can be interpreted as not being selected by your model.

• But you can also include extra information. For example, do your data features have a natural grouped structure? Like when dealing with genetic datasets where variables can be grouped into genetical pathways, or like when dealing with econometrics datasets where variables can be grouped in terms of geographical information, industrial sector etc? In these kind of situations penalizations such as group lasso / sparse group lasso can achieve really good results.

• There are many other regularizations such as SCAD or adaptive LASSO among others.

Answer 2

You know already, I believe, that in machine learning feature selection is almost always omitted. Actually, if you are asked to do it, it is to save computational resources rathen than to enhance predictions at higher computational cost.

Would we do that if we had "infinite computation power"? Well, probably yes, but for the way ML works in our finite-resources world, that wouldn't really change much.

As every statistician knows well, useless features do exist. You often have variables in your data set that simply don't have any effect on the outcome, there is no relation, it's just noise that will creep into predictions and worsen them.

However, let's see how this works in a machine learning workflow:

To measure how feature selection benefits your model, you have to implement it as an algorithm and try it on a training set, to compare the results on a developement set (or many, if you do CV). You have to chain the feature selection algorithm with the model learning one, which uses at least one form of regularization, probably two or even some more. This way you have another hyper-parameter to tune in your pipeline: I'm not going into feature selection algorithms, but they need a parameter that specifies how strict must the selection be, or at least to dictate if it shall run or not. Adding a parameter to tune makes the training algorithm more flexible, which easily means better developement set results, but also less reliable ones: tuning is a kind of learning itself, and it can overfit.

Also, feature selection algorithms are either linear (which means that they can leave out useful non-linear predictors) or unstable by some other mean (like random forest feature importance, which can be computed in different ways, each of them has pros and cons), or simply too expensive (like trying to run the entire model with certain, if not all, subsets of the available predictors). Even if we don't worry about computation time and go for the most expensive method possible, regularization already does a pretty good job on reducing the effect of noisy variables, so dropping them can only improve the model by a tiny bit, and still it can worsen it if the variable actually is of some use. Cross validation is not an exact method, it can select a bad feature selector.

There are such better ways to use that computational power!