How do you find the best feature combination to achieve maximum accuracy?

Question

I have a set of 9 possible features that I can extract from an image. Does anyone know of a method that I can employ to utilize the best feature combination to get the maximum possible classification accuracy?

Answer 1

It is quite possible that best performance is gained by using regularisation (e.g. ridge regression) rather than feature selection. When performing feature selection it is all too easy to over-fit the feature selection criterion and end up with a model that performs poorly. With regularisation you only have one regularisation parameter to estimate, and so fewer (effective) degrees of freedom and hence are less likelu to overfit the model selection criterion. Having said which with only 9 attributes this shouldn't be too much of a problem if you have a large number of examples.

The key point I am making is that don't assume that feature selection will give you better performance. Often it doesn't and sometimes it makes performance worse. If generalisation performance is the improtant factor, then try regularsation instead. This is essentially the advice that Millar gives in his monograph on "subset selection in regression".