# Tutorials for feature engineering

As is known to all, feature engineering is extremely important to machine learning, however I found few materials associated with this area. I participated to several competitions in Kaggle and believe that good features may even be more important than a good classifier in some cases. Does anyone know any tutorials about feature engineering, or is this pure experience?

• Do you mean pre-processsing of features (normalization and other transformations) or feature selection? – MattBagg Nov 11 '12 at 22:25
• @mb3041023 No, the step prior to both, in which you convert some raw data like texts, images or series into some usable attributes. – user88 Nov 12 '12 at 0:46
• In my experience, a huge part of the problem of machine learning, is literally setting up the correct problem to be solved/optimized (i.e. features, feature representation, selection, etc). I'd love to see a book purely dedicated to empirical feature selection and pre-processing with many real life illustrations (like kaggle). If anyone knows of one, pls. post. There are several books dedicated to things like data cleaning/data imputation, but a dedicated practical text on feature selection is sorely needed. – pat Nov 12 '12 at 1:19
• Take a look at: "Feature Extraction: Foundations and Applications", 2006 – jasonb Mar 4 '13 at 21:57
• @jasonb, how about author, size, price, and a link, something like this: Guyon ed., Feature Extraction: Foundations and Applications 2006, 778p, $306 – denis Dec 24 '13 at 15:10 ## 1 Answer I would say experience -- basic ideas are: • to fit how classifiers work; giving a geometry problem to a tree, oversized dimension to a kNN and interval data to an SVM are not a good ideas • remove as much nonlinearities as possible; expecting that some classifier will do Fourier analysis inside is rather naive (even if, it will waste a lot of complexity there) • make features generic to all objects so that some sampling in the chain won't knock them out • check previous works -- often transformation used for visualisation or testing similar types of data is already tuned to uncover interesting aspects • avoid unstable, optimizing transformations like PCA which may lead to overfitting • experiment a lot • How do you define "interval data"? I searched on Google and found many different definitions. – power Nov 12 '12 at 3:16 • can you elaborate on the PCA point? – Daniel Velkov Nov 12 '12 at 23:16 • @power For instance set like$x$and decision$|x-\text{nearest prime}|<0.3\$, i.e. when attribute should be splitted in many intervals rather than put in some simple continuous transformation. – user88 Nov 13 '12 at 8:07
• @DanielVelkov When you bootstrap PCA on a rather noisy data the components are often unstable; this promotes the idea to make one global PCA on the whole available set, what leaks information and is a straight way to spoil the evaluation. – user88 Nov 15 '12 at 13:32
• @mbq what if PCA is run only on the training set, the way it's supposed to be? – Daniel Velkov Nov 15 '12 at 20:28