It is generally accepted that the most important factor for successful machine learning is quality feature engineering:

Feature Engineering is the Key

At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used ... Feature engineering is more difficult [than learning] because it is domain-specific, while learners can be largely general purpose.

Pedro Domingos, "A Few Useful Things to Know About Machine Learning", Communications of the ACM (CACM), Vol. 55, No. 10, Pages 78-87, Oct 2012.

So I was searching for academic references on the subject of "Feature Quality Criteria", yet I had little success. Has anyone came across a credible paper or text-book that elaborates on general criteria for feature quality?

What intuitively comes to my mind is the discriminative and expressive power of features. Of course one might ask: "What do you mean by discriminative and expressive power of features? Explain them formally" ...

This is exactly why I'm searching for credible references.


Part IV in Applied Predictive Modelling by Max Kuhn and Kjell Johnson addresses - besides others - features, their importance, and related aspects. Though this might not be the single perfect reference (especially concerning the "formal" criterion), this chapter discusses the underlying issues regarding discriminative/expressive power with simple examples, e.g. by demonstrating the negative effect of random features.

BTW: Max Kuhn might have other useful literature on those aspects too. For example, he pointed out the importance of features over models e.g. in his 2015 overview of predictive modelling (around slide 50).

  • $\begingroup$ I checked out Part-IV of the reference, and the only relevant chapter seemed to be "Chapter-19, Introduction to Feature Selection". Yet the content of that chapter is about various feature selection methods and not "feature engineering". There is very little discussion on "what are good features" and the majority of the chapter is introducing different feature selection methods -- Thank you anyway. $\endgroup$ – Seyed Mohammad May 21 '16 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.