# What is the most efficient feature engeenering approach for Kaggle-like data?

I am wondering what could be the most efficient approach to feature engineering with data consisting of huge number of anonymous variables of unknown, or blurry meaning? To not make it too broad, let's focus on numerical and one-hot-encoded categorical variables, and ignore working with textual data. By efficient I mean here such approach that could possibly lead to greatest improvement out-of-the box and is unlikely to do harm. Obviously there is lots of things that could be done with such data and lots of ways how you could waste your time producing literally hundreds of meaningless features (say polynomials of all the features and their interactions), so I'm asking for the approach that is unlikely for you to lead to wasting your time, just to get very minor improvements in the predictions quality. Is there any safe recommendation for data like this? I am not looking for opinions, but for something that has proven accuracy.

• Great question, as a part time Kaggler I might even have an idea which data you may be talking about :-) For interested: kaggle.com/c/porto-seguro-safe-driver-prediction – Łukasz Grad Oct 29 '17 at 13:44
• Nice question! I think a lot of people have pounder about it for a lot of time! (+1) – usεr11852 says Reinstate Monic Oct 29 '17 at 14:14
• @ŁukaszGrad actually I did not have any particular dataset, or even Kaggle itself, in mind, I was actually thinking of a general problem, since everyone seems to start and end the discussion about feature engineering by saying that it is an "art" without discussing what actually has a proven effect. – Tim Oct 29 '17 at 14:17

Going to a bit more detail, one might argue about transforming categorical features to numerical features being extremely helpful. Indeed it can be tremendously helpful when dealing with categorical variables of high cardinality (eg. see all the options that catboost offers); I fully agree with that idea. Nevertheless these techniques is not necessarily better than one-hot encoding when cardinality is not high or when one uses binary-encoding (eg. see here a more careful investigation); ie. they do not constitute an approach with "proven accuracy".
Similarly, normalising numerical features to have a specific range (eg. $[-1,+1]$) or distribution (eg. $\sim N(0,1)$) is a necessity when working with regularisation methods like ridge or LASSO regression but realistically it will make little difference to random forests and other tree-based learners who are mostly using ordering information. Again, no guarantee that normalisation is not just a waste of time.