I am wondering what could be the most efficient approach to feature engineering with data consisting of huge number of anonymous variables of unknown, or blurry meaning? To not make it too broad, let's focus on numerical and one-hot-encoded categorical variables, and ignore working with textual data. By efficient I mean here such approach that could possibly lead to greatest improvement out-of-the box and is unlikely to do harm. Obviously there is lots of things that could be done with such data and lots of ways how you could waste your time producing literally hundreds of meaningless features (say polynomials of all the features and their interactions), so I'm asking for the approach that is unlikely for you to lead to wasting your time, just to get very minor improvements in the predictions quality. Is there any safe recommendation for data like this? I am not looking for opinions, but for something that has proven accuracy.

  • $\begingroup$ Great question, as a part time Kaggler I might even have an idea which data you may be talking about :-) For interested: kaggle.com/c/porto-seguro-safe-driver-prediction $\endgroup$ – Łukasz Grad Oct 29 '17 at 13:44
  • $\begingroup$ Nice question! I think a lot of people have pounder about it for a lot of time! (+1) $\endgroup$ – usεr11852 says Reinstate Monic Oct 29 '17 at 14:14
  • $\begingroup$ @ŁukaszGrad actually I did not have any particular dataset, or even Kaggle itself, in mind, I was actually thinking of a general problem, since everyone seems to start and end the discussion about feature engineering by saying that it is an "art" without discussing what actually has a proven effect. $\endgroup$ – Tim Oct 29 '17 at 14:17

Somewhat anticlimactically, I think that the most efficient as well as universal feature engineering approach is domain expertise; pretty much everything is second to that. That said, a dominant second approach is using no feature engineering at all.

Going to a bit more detail, one might argue about transforming categorical features to numerical features being extremely helpful. Indeed it can be tremendously helpful when dealing with categorical variables of high cardinality (eg. see all the options that catboost offers); I fully agree with that idea. Nevertheless these techniques is not necessarily better than one-hot encoding when cardinality is not high or when one uses binary-encoding (eg. see here a more careful investigation); ie. they do not constitute an approach with "proven accuracy".

Similarly, normalising numerical features to have a specific range (eg. $[-1,+1]$) or distribution (eg. $\sim N(0,1)$) is a necessity when working with regularisation methods like ridge or LASSO regression but realistically it will make little difference to random forests and other tree-based learners who are mostly using ordering information. Again, no guarantee that normalisation is not just a waste of time.

That's why I argue for domain expertise. Notice domain expertise does not need to be vast rather enough to guarantee "common sense" within the domain of application. On certain specific tasks there might be features that are worth always testing. For example, for image recognition, speech recognition and time-series analysis, edge-detection, spectral decomposition and change-point analysis respectively are obvious and most likely helpful features to augment a training set.

So... is domain expertise an out-of-the-box approach? I would argue yes but if we convenience ourselves for no then our best feature engineering option is using no feature engineering at all. That means using learners that are very flexible and can learn strong non-linear associations themselves. Gradient boosting machines and (deep) neural networks stand out as immediate examples. Clearly we now pick the box, but hey, it is an out-of-the-box choice after all. :)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.