SVM and non-linear predictive models - feature selection

Question

Just throwing out a general question. What do people think of applying feature selection methods when using SVMs to build predictive models? I understand that SVM have built in regularization with how they're trained, but I've heard for certain cases (e.g. features >> examples) feature selection is still useful. I'm mainly talking about non-linear SVMs (e.g. kernelized).

Is there a difference in effectiveness of feature selection when say # features >> # examples vs. # features ~ # examples vs. # features << examples?

If you don't think feature selection is useful, please explain why. If you think feature seleciton is useful (and have practical experience in this regard), please let us know what methods you've used and how they worked.

Also open to entirely different approaches to non-linear predictive modeling other than SVMs.

Thanks!

Answer 1

I'd say that it is generally a bad idea to use feature selection for non-linear support vector machines. The reason for using a maximal margin classifier is that it is an approximate implementation of a theoretical performance bound that is independent of the dimensionality of the feature space in which it is constructed. This means that over-fitting can be controlled via structural risk minimisation by carefully selecting the regularisation parameter, $C$. HOWEVER, and it is a rather big "however", this bit of computational learning theory only applies to the construction of the maximal margin classifier in a feature space induced by a fixed kernel. Any attempt to tune the kernel function, e.g. by adapting the kernel parameters, or via feature selection will invalidate the bounds on performance and in doing so potentially circumvent the structural risk minimisation principle.

The other reason why it is generally a bad idea is that feature selection is difficult. If you perform feature selection by minimising some statistic computed on a finite sample of data, it is all too easy to over-fit the selection criterion, and end up with a worse model that you started with. This is especially true where there are many degrees of freedom in the optimisation problem (e.g. one binary parameter for the inclusion of each feature). The use of regularisation instead gives one (continuous) degree of freedom, which makes over-fitting more difficult.

I have been performing some experiments for feature selection for non-linear kernel machines, and the basic message is that in general efforts at feature selection will result in lower generalisation performance. It helps on some datasets (sometimes it helps a lot), but usually it makes things worse (sometimes much worse).

Answer 2

The whole point of deploying a non-linear kernel is to increase the dimensionality of the space, hoping that in the high dimensional space, there would be a linear solution to the problem.

If you have lot of features, hopefully, they will offer some little extra information that can be used to build a better model. Doing feature selection, i.e. killing original features, and then deploying non-linear kernel, i.e. creating artificial features by combinations of the leftovers, is... strange.

Not sure about your problem but in practice it may help if the features you filter out are not offering any extra information to the solution, e.g. may happen if they are so correlated that indeed all useful information exists in remaining features. In practice feature selection may help by reducing the training time required (probably far easier and practical to go for linear kernel).

PS. and as I was writing this I realised how old is this question :)