3
$\begingroup$

I have a simple but difficult question. Does feature selection before SVM help? I have a data set that has ~1100 features but a lot of these are redundant data / uncorrelated data. Can someone give me a reason why it should / should not help? Thanks a lot.

$\endgroup$
  • $\begingroup$ user27525 - Welcome to CV. Your question is very broad and would appear to require an answer based on opinion. Can you edit to make your question more specific and in such a way that the answer could be supported with facts or references? Please see the help on asking a good question and see if you can edit to improve it. $\endgroup$ – Glen_b Jul 2 '13 at 0:33
  • $\begingroup$ In case you are interested in SVM-based recursive feature elimination, see this thread, for example. $\endgroup$ – chl Jan 29 '14 at 16:00
9
$\begingroup$

The SVM is an approximate implementation of a theoretical bound on the generalisation performance that is independent of the dimensionality of the feature space. This means that there is a good reason to suggest that performing feature selection might not make the performance of the classifier any better.

The reason that the SVM works is because it uses regularisation (like ridge regression) to avoid over-fitting, so provided you set the regularisation parameter $C$ properly (e.g. using cross-validation), the performance ought to be good without feature selection.

The thing that is often not mentioned about feature selection is that it can easily make performance worse. The reason for this is that the more choices about the model that are made by optimising some statistic evaluated over the training sample, the more likely you are to over-fit the training sample, and feature selection often ends up making many more choices about the model (worst case $2^d$ where $d$ is the number of parameters). In his monograph on feature subset selection for regression[0], Millar suggests that if you are primarily interested in generalisation performance, then use ridge regression instead and don't do any feature selection. This is in accord with my experience, I think the reason is that it is more difficult to over-fit with one continuous parameter tuned using cross-validation than choosing the best of the $2^n$ combinations of features.

  • [0] Millar, A. (2002). Subset Selection in Regression, Second Editon. Chapman & Hall/CRC Monographs on Statistics & Applied Probability.
$\endgroup$
2
$\begingroup$

From your quote " have a data set that has ~1100 features but a lot of these are redundant data / uncorrelated data" I would strongly recommend feature reduction. Jacob is right that it won't improve the accuracy (in terms of explained variance) in-sample (it will do the opposite) but it will improve reliability of the model out of sample. If you start fitting this type of data raw straight out of the box you are unlikely to come up with anything useful.

The SVM itself is nothing more than an optimization on a linear boundary, what makes it powerful are transformations like kernel methods, PCA, etc. that produce data and models that make sense. Have good think about how you want to represent the structure in your data (perhaps first do some unsupervised learning on the raw data to discover structure). And no one can really tell you how to reduce the dimensionality (i.e. extract the salient features) of your problem without actually looking at the data.

Once you have made a dent in the feature reduction make sure to do things like cross-validation and random sampling to avoid overfitting.

$\endgroup$
  • 1
    $\begingroup$ "but it will improve reliability of the model out of sample" - this is incorrect, it is very easy to make generalisation performance worse by over-fitting the choice of features. Also regularisation is what makes the SVM a good tool, it doesn't actually work any better than ridge regression. The last piece of advice is very dangerous, you need to perform the feature selection separately in each fold of the cross-validation to avoid bias (pnas.org/content/99/10/6562.abstract). $\endgroup$ – Dikran Marsupial Apr 5 '14 at 19:09
  • 1
    $\begingroup$ It all depends on whether you use data reduction (i.e., masked to $Y$) or $Y$-association-driven feature selection (which causes all the problems mentioned above). $\endgroup$ – Frank Harrell Apr 5 '14 at 19:27
0
$\begingroup$

Reducing the number of features will surely reduce the computational complexity of your model. Will it improve the accuracy of your model? Perhaps, but not necessarily. It really depends on what the data you are given looks like.

$\endgroup$
  • 3
    $\begingroup$ No, you would be undoing all the advantages of svm to do pre-svm feature reduction, at least if such reduction is unblinded to $Y$. $\endgroup$ – Frank Harrell Jul 2 '13 at 18:34
  • 1
    $\begingroup$ Redundant features/dim reduction: No. Consider the case of dropping the 0-weighted loadings of PCA before computing the gram matrix. It has absolutely no effect. Redundant features have no effect on the characteric of the Gram matrix for any mercer kernel. If the features added rank, it'd be different. If the features added rank and were correlated to the outcome somehow, then it'd be useful to keep them. If they added rank but were uncorrelated, it'd be like adding noise to the Gram. $\endgroup$ – Jessica Mick Oct 1 '13 at 4:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.