I have one question with respect to need to use feature selection methods (Random forests feature importance value or Univariate feature selection methods etc) before running a statistical learning algorithm.

We know to avoid overfitting we can introduce regularization penalty on the weight vectors.

So if I want to do linear regression, then I could introduce the L2 or L1 or even Elastic net regularization parameters. To get sparse solutions, L1 penalty helps in feature selection.

Then is it still required to do feature selection before Running L1 regularizationn regression such as Lasso?. Technically Lasso is helping me reduce the features by L1 penalty then why is the feature selection needed before running the algo?

I read a research article saying that doing Anova then SVM gives better performance than using SVM alone. Now question is: SVM inherently does regularization using L2 norm. In order to maximise the margin, it is minimising the weight vector norm. So it is doing regularization in it's objective function. Then technically algorithms such as SVM should not be bothered about feature selection methods?. But the report still says doing Univariate Feature selection before normal SVM is more powerful.

Anyone with thoughts?

  • $\begingroup$ One question I would raise is how does SVM regularisation work with kernel methods? L2 norm reg relies on inputs being normalised. However if you use a kernel (eg polynomial) then your effective inputs (ie the polynomial powers) are no longer normalised. $\endgroup$
    – seanv507
    May 5, 2015 at 13:52

5 Answers 5


Feature selection sometimes improves the performance of regularized models, but in my experience it generally makes generalization performance worse. The reason for this is that the more choices we make regarding our model (including the values of the parameters, the choice of features, the setting of hyper-parameters, the choice of kernel...), the more data we need to make these choices reliably. Generally we make these choices by minimizing some criterion evaluated over a finite set of data, which means that the criterion inevitably has a non-zero variance. As a result, if we minimize the criterion too agressively, we can over-fit it, i.e. we can make choices that minimize the criterion because of features that depend on the particular sample on which it is evaluated, rather than because it will produce a genuine improvement in performance. I call this "over-fitting in model selection" to differentiate it from the more familiar form of over-fitting resulting from tuning the model parameters.

Now the SVM is an approximate implementation of a bound on generalization performance that does not depend on the dimensionality, so in principle, we can expect good performance without feature selection, provided the regularization parameters are correctly chosen. Most feature selection methods have no such performance "guarantees".

For L1 methods, I certainly wouldn't bother with feature selection, as the L1 criterion is generally effective in trimming features. The reason that it is effective is that it induces an ordering in which features enter and leave the model, which reduces the number of available choices in selecting features, and hence is less prone to over-fitting.

The best reason for feature selection is to find out which features are relevant/important. The worst reason for feature selection is to improve performance, for regularised models, generally it makes things worse. However, for some datasets, it can make a big difference, so the best thing to do is to try it and use a robust, unbiased performance evaluation scheme (e.g. nested cross-validation) to find out whether yours is one of those datasets.

  • $\begingroup$ What do you mean by nested cross-valdiation? Does it applying k-fold cross validation N times? $\endgroup$ May 6, 2015 at 14:51

A lot of people do think that regularization is enough to take care of extraneous variables and no variable selection is needed if you appropriately regularize, do partial pooling, create hierarchical models, etc. when the goal is predictive accuracy. For example, if a parameter estimate for a particular variable $j$ is regularized all the way down to $\hat{\beta}_j = .0001$ or is entirely removed from the model ($\hat{\beta}_j = 0$) really doesn't make a big difference in prediction problems.

However, there are still reasons to completely remove a variable.

  1. Sometimes the goal is not predictive accuracy but explanation of a world phenomenon. Sometimes you want to know what variables do and do not impact a certain dependent variable. In these types of situations, a parsimonious model is a preferred way to understand and interpret
  2. You're setting yourself up for risk. If you leave in a variable $\hat{\beta}_j$ that truly doesn't have an effect then you're setting yourself up for the possibility that if you collect different data then the variable impacts results. This is especially pertinent for models that get refit over and over again with different applications.
  3. Computational reasons - a model with fewer variables generally runs faster and you don't have to store the data for those extraneous variables.
  • 3
    $\begingroup$ Hi TrynnaDoStat, i totally agree with your point 1 and 3. But argument 2 does not really hit the point. If you do feature selection with the same set of data, you also have the risk to have choosen the wrong feature set. Because one random variable might seems to correlated well with the goal variable in the data. $\endgroup$
    – gstar2002
    Jan 18, 2017 at 18:29
  • $\begingroup$ I suppose what I'm saying with point 2 is that if you refit a model over and over again (let's say 100s of times) every once in a while you're going to get a $\hat{\beta}_j$ that has a large enough value to impact results. Even though the vast majority of the time you get a tiny value. I realize this is a little contrived any may not happen very often in the real world. $\endgroup$ Jan 19, 2017 at 0:09

I don't think overfitting is the reason that we need feature selection in the first place. In fact, overfitting is something that happens if we don't give our model enough data, and feature selection further reduces the amount of data that we pass our algorithm.

I would instead say that feature selection is needed as a preprocessing step for models which do not have the power to determine the importance of features on their own, or for algorithms which get much less efficient if they have to do this importance weighting on their own.

Take for instance a simple k-nearest neighbor algorithm based on Euclidean distance. It will always look at all features as having the same weight or importance to the final classification. So if you give it 100 features but only three of these are relevant for your classification problem, then all the noise from these extra features will completely drown out the information from the three important features, and you won't get any useful predictions. If you instead determine the critical features beforehand and pass only those to the classifier, it will work much better (not to mention be much faster).

On the other hand, look at a random forest classifier. While training, it will automatically determine which features are the most useful by finding an optimal split by choosing from a subset of all features. Therefore, it will do much better at sifting through the 97 useless features to find the three good ones. Of course, it will still run faster if you do the selection beforehand, but its classification power will usually not suffer much by giving it a lot of extra features, even if they are not relevant.

Finally, look at neural networks. Again, this is a model which has the power to ignore irrelevant features, and training by backpropagation will usually converge to using the interesting features. However, it is known that standard training algorithm converge much faster if the inputs are "whitened", i.e., scaled to unit variance and with removed cross correlation (LeCun et al, 1998). Therefore, although you don't strictly need to do feature selection, it can pay in pure performance terms to do preprocessing of the input data.

So in summary, I would say feature selection has less to do with overfitting and more with enhancing the classification power and computational efficiency of a learning method. How much it is needed depends a lot on the method in question.

  • 3
    $\begingroup$ (1) I don't agree with your first proposition. Feature selection doesn't reduce amount of data but reduces number of features. The number of instances (samples) remains the same, and it can help to overfitting because, the classifier needs fewer parameters (if it is a parametric model) to fit the data. Fewer parameters mean less representation power, so less likely to overfit. (2) What is the type of feature selection algorithm you mentioned to use before KNN? How it knows which features will be more effective? $\endgroup$ May 6, 2015 at 15:10
  • 1
    $\begingroup$ @yasin.yazici: Say you're doing handwritten digit classification and you throw away all but the upper leftmost pixel. Didn't you just reduce the amount of data you had? $\endgroup$
    – cfh
    May 6, 2015 at 18:29
  • $\begingroup$ @chf No, it only reduces number of features. Lest say dataset is MNIST there are 784 features for each sample and 60000 samples. If you throw away some part of your features, you still have 60000 samples. $\endgroup$ May 6, 2015 at 18:47
  • 1
    $\begingroup$ @yasin.yazici: My point is, number of samples is not the same as amount of data. Data is the whole matrix of "samples x features". $\endgroup$
    – cfh
    May 6, 2015 at 20:53
  • $\begingroup$ Feature selection can (and often will) overfit. If you are using regularized learners and don't care about feature importance, then, unless you have lots of data and a robust validation scheme, I don't see much use for feature selection. $\endgroup$
    – Firebug
    Jun 27, 2016 at 14:15

In the case of lasso, preprocessing the data to remove nuisance features is actually pretty common. For a recent paper discussing ways to do this, please see Xiang et al's Screening Tests for Lasso Problems. The common motivation mentioned in the papers I've seen is to reduce the computational burden of computing the solution.


I think if you do not have sufficient number of data points to robustly optimize the parameters you can do feature selection to remove some variables. But I would not suggest doing too much of it since you can lose the signal you want to model.

Plus there might be certain features you do not want in your models based on business understanding which you may want to remove.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.