Do we still need to do feature selection while using Regularization algorithms?

Question

I have one question with respect to need to use feature selection methods (Random forests feature importance value or Univariate feature selection methods etc) before running a statistical learning algorithm.

We know to avoid overfitting we can introduce regularization penalty on the weight vectors.

So if I want to do linear regression, then I could introduce the L2 or L1 or even Elastic net regularization parameters. To get sparse solutions, L1 penalty helps in feature selection.

Then is it still required to do feature selection before Running L1 regularizationn regression such as Lasso?. Technically Lasso is helping me reduce the features by L1 penalty then why is the feature selection needed before running the algo?

I read a research article saying that doing Anova then SVM gives better performance than using SVM alone. Now question is: SVM inherently does regularization using L2 norm. In order to maximise the margin, it is minimising the weight vector norm. So it is doing regularization in it's objective function. Then technically algorithms such as SVM should not be bothered about feature selection methods?. But the report still says doing Univariate Feature selection before normal SVM is more powerful.

Anyone with thoughts?

Answer 1

Feature selection sometimes improves the performance of regularized models, but in my experience it generally makes generalization performance worse. The reason for this is that the more choices we make regarding our model (including the values of the parameters, the choice of features, the setting of hyper-parameters, the choice of kernel...), the more data we need to make these choices reliably. Generally we make these choices by minimizing some criterion evaluated over a finite set of data, which means that the criterion inevitably has a non-zero variance. As a result, if we minimize the criterion too agressively, we can over-fit it, i.e. we can make choices that minimize the criterion because of features that depend on the particular sample on which it is evaluated, rather than because it will produce a genuine improvement in performance. I call this "over-fitting in model selection" to differentiate it from the more familiar form of over-fitting resulting from tuning the model parameters.

Now the SVM is an approximate implementation of a bound on generalization performance that does not depend on the dimensionality, so in principle, we can expect good performance without feature selection, provided the regularization parameters are correctly chosen. Most feature selection methods have no such performance "guarantees".

For L1 methods, I certainly wouldn't bother with feature selection, as the L1 criterion is generally effective in trimming features. The reason that it is effective is that it induces an ordering in which features enter and leave the model, which reduces the number of available choices in selecting features, and hence is less prone to over-fitting.

The best reason for feature selection is to find out which features are relevant/important. The worst reason for feature selection is to improve performance, for regularised models, generally it makes things worse. However, for some datasets, it can make a big difference, so the best thing to do is to try it and use a robust, unbiased performance evaluation scheme (e.g. nested cross-validation) to find out whether yours is one of those datasets.

Answer 2

A lot of people do think that regularization is enough to take care of extraneous variables and no variable selection is needed if you appropriately regularize, do partial pooling, create hierarchical models, etc. when the goal is predictive accuracy. For example, if a parameter estimate for a particular variable $j$ is regularized all the way down to $\hat{\beta}_j = .0001$ or is entirely removed from the model ($\hat{\beta}_j = 0$) really doesn't make a big difference in prediction problems.

However, there are still reasons to completely remove a variable.

1. Sometimes the goal is not predictive accuracy but explanation of a world phenomenon. Sometimes you want to know what variables do and do not impact a certain dependent variable. In these types of situations, a parsimonious model is a preferred way to understand and interpret
2. You're setting yourself up for risk. If you leave in a variable $\hat{\beta}_j$ that truly doesn't have an effect then you're setting yourself up for the possibility that if you collect different data then the variable impacts results. This is especially pertinent for models that get refit over and over again with different applications.
3. Computational reasons - a model with fewer variables generally runs faster and you don't have to store the data for those extraneous variables.

Answer 3

I don't think overfitting is the reason that we need feature selection in the first place. In fact, overfitting is something that happens if we don't give our model enough data, and feature selection further reduces the amount of data that we pass our algorithm.

I would instead say that feature selection is needed as a preprocessing step for models which do not have the power to determine the importance of features on their own, or for algorithms which get much less efficient if they have to do this importance weighting on their own.

Take for instance a simple k-nearest neighbor algorithm based on Euclidean distance. It will always look at all features as having the same weight or importance to the final classification. So if you give it 100 features but only three of these are relevant for your classification problem, then all the noise from these extra features will completely drown out the information from the three important features, and you won't get any useful predictions. If you instead determine the critical features beforehand and pass only those to the classifier, it will work much better (not to mention be much faster).

On the other hand, look at a random forest classifier. While training, it will automatically determine which features are the most useful by finding an optimal split by choosing from a subset of all features. Therefore, it will do much better at sifting through the 97 useless features to find the three good ones. Of course, it will still run faster if you do the selection beforehand, but its classification power will usually not suffer much by giving it a lot of extra features, even if they are not relevant.

Finally, look at neural networks. Again, this is a model which has the power to ignore irrelevant features, and training by backpropagation will usually converge to using the interesting features. However, it is known that standard training algorithm converge much faster if the inputs are "whitened", i.e., scaled to unit variance and with removed cross correlation (LeCun et al, 1998). Therefore, although you don't strictly need to do feature selection, it can pay in pure performance terms to do preprocessing of the input data.

So in summary, I would say feature selection has less to do with overfitting and more with enhancing the classification power and computational efficiency of a learning method. How much it is needed depends a lot on the method in question.

Answer 4

I think if you do not have sufficient number of data points to robustly optimize the parameters you can do feature selection to remove some variables. But I would not suggest doing too much of it since you can lose the signal you want to model.

Plus there might be certain features you do not want in your models based on business understanding which you may want to remove.

Answer 5

In the case of lasso, preprocessing the data to remove nuisance features is actually pretty common. For a recent paper discussing ways to do this, please see Xiang et al's Screening Tests for Lasso Problems. The common motivation mentioned in the papers I've seen is to reduce the computational burden of computing the solution.