Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a set of 9 possible features that I can extract from an image. Does anyone know of a method that I can employ to utilize the best feature combination to get the maximum possible classification accuracy?

share|improve this question
This is a very application-specific problem. You will need to provide more details on the actual problem of interest. – cardinal May 1 '12 at 13:15
binary classification, continuous features ? A (linear) SVM will give you 9 feature weights + offset, quick, easy and often good in practice. However SVMs don't optimize classification accuracy per se; also, there are various loss functions for misclassifications. – denis May 2 '12 at 15:53

It is quite possible that best performance is gained by using regularisation (e.g. ridge regression) rather than feature selection. When performing feature selection it is all too easy to over-fit the feature selection criterion and end up with a model that performs poorly. With regularisation you only have one regularisation parameter to estimate, and so fewer (effective) degrees of freedom and hence are less likelu to overfit the model selection criterion. Having said which with only 9 attributes this shouldn't be too much of a problem if you have a large number of examples.

The key point I am making is that don't assume that feature selection will give you better performance. Often it doesn't and sometimes it makes performance worse. If generalisation performance is the improtant factor, then try regularsation instead. This is essentially the advice that Millar gives in his monograph on "subset selection in regression".

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.