As is known to all, feature engineering is extremely important to machine learning, however I found few materials associated with this area. I participated to several competitions in Kaggle and believe that good features may even be more important than a good classifier in some cases. Does anyone know any tutorials about feature engineering, or is this pure experience?

  • 1
    $\begingroup$ Do you mean pre-processsing of features (normalization and other transformations) or feature selection? $\endgroup$ – MattBagg Nov 11 '12 at 22:25
  • 2
    $\begingroup$ @mb3041023 No, the step prior to both, in which you convert some raw data like texts, images or series into some usable attributes. $\endgroup$ – user88 Nov 12 '12 at 0:46
  • 8
    $\begingroup$ In my experience, a huge part of the problem of machine learning, is literally setting up the correct problem to be solved/optimized (i.e. features, feature representation, selection, etc). I'd love to see a book purely dedicated to empirical feature selection and pre-processing with many real life illustrations (like kaggle). If anyone knows of one, pls. post. There are several books dedicated to things like data cleaning/data imputation, but a dedicated practical text on feature selection is sorely needed. $\endgroup$ – pat Nov 12 '12 at 1:19
  • 2
    $\begingroup$ Take a look at: "Feature Extraction: Foundations and Applications", 2006 $\endgroup$ – jasonb Mar 4 '13 at 21:57
  • 2
    $\begingroup$ @jasonb, how about author, size, price, and a link, something like this: Guyon ed., Feature Extraction: Foundations and Applications 2006, 778p, $306 $\endgroup$ – denis Dec 24 '13 at 15:10

I would say experience -- basic ideas are:

  • to fit how classifiers work; giving a geometry problem to a tree, oversized dimension to a kNN and interval data to an SVM are not a good ideas
  • remove as much nonlinearities as possible; expecting that some classifier will do Fourier analysis inside is rather naive (even if, it will waste a lot of complexity there)
  • make features generic to all objects so that some sampling in the chain won't knock them out
  • check previous works -- often transformation used for visualisation or testing similar types of data is already tuned to uncover interesting aspects
  • avoid unstable, optimizing transformations like PCA which may lead to overfitting
  • experiment a lot
  • $\begingroup$ How do you define "interval data"? I searched on Google and found many different definitions. $\endgroup$ – power Nov 12 '12 at 3:16
  • $\begingroup$ can you elaborate on the PCA point? $\endgroup$ – Daniel Nov 12 '12 at 23:16
  • $\begingroup$ @power For instance set like $x$ and decision $|x-\text{nearest prime}|<0.3$, i.e. when attribute should be splitted in many intervals rather than put in some simple continuous transformation. $\endgroup$ – user88 Nov 13 '12 at 8:07
  • $\begingroup$ @DanielVelkov When you bootstrap PCA on a rather noisy data the components are often unstable; this promotes the idea to make one global PCA on the whole available set, what leaks information and is a straight way to spoil the evaluation. $\endgroup$ – user88 Nov 15 '12 at 13:32
  • $\begingroup$ @mbq what if PCA is run only on the training set, the way it's supposed to be? $\endgroup$ – Daniel Nov 15 '12 at 20:28

There is a book from O'Reilly called "Feature Engineering for Machine Learning" by Zheng et al.

I read the book and it covers different types of data (e.g categorical, text...) and describes different aspects of feature engineering that go with it. This includes things like normalization of data, feature selection, tf-idf in text.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.