Feature engineering is often an important component to machine learning (it was used heavily to win the KDD Cup in 2010). However, I find that most feature engineering techniques either

  • destroy any intuitive meaning of the underlying features or
  • are very specific to a particular domain or even particular types of features.

A classic example of the former would be principal component analysis. It would seem to me that any knowledge that a subject-matter expert would have about the features would be destroyed by converting those features to principal components.

Contrast that with a simple technique of converting a date into features for "day of month" and "day of week." The underlying meaning is still retained in the new features, but obviously this particular technique only applies for dates and not arbitrary features.

Is there any standard body of feature engineering techniques that do not destroy the meaning of the underlying features while also being applicable to arbitrary domains (or at least a wide variety of domains)?

  • 2
    $\begingroup$ PCA can sometimes be used to find intuitive meanings for the features - e.g. eigenfaces. $\endgroup$
    – tdc
    Feb 14, 2012 at 13:10
  • $\begingroup$ Can you give (more) examples of the data you have in mind ? if you can be more specific about your application (even some arbitrary example) it will be easier to give (more) accurate answer. $\endgroup$
    – Dov
    Feb 17, 2012 at 18:18
  • 1
    $\begingroup$ @Dov Well the whole point is that (ideally) I'd like something that could work for just about any structured, tabular dataset (one that has datapoints and features). So this could be sales data, financial data, drug discovery data, baseball data, etc. $\endgroup$ Feb 17, 2012 at 19:28

2 Answers 2


I am aware of one decomposition method (but maybe there are more...) that can be useful in a scenarios like you describe. It is like 2D-PCA - a high order decomposition method where the decomposition (i.e the factors) have some meaning. You can see examples and read about it here and here and try to here

  • $\begingroup$ + please forgive me that I am not a native English speaker :) $\endgroup$
    – Dov
    Feb 8, 2012 at 6:34
  • $\begingroup$ From what I was told, last step of PCA should be attempt to find meaning for principial componenet. $\endgroup$
    – jb.
    Feb 8, 2012 at 19:48

Recent deep learning methods using Restricted Boltzmann Machine have shown nice features on several data types (audio, images, text).

Since these methods create a generative model, you can often generate really nice samples from the model.

Check out Hinton's publications. http://www.cs.toronto.edu/~hinton/

These methods aren't totally general (run same code on every data), but the underlying model is usually similar.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.