does feature engineering matter when doing Random Forest or Gradient Boosting?

Question

For linear models (such as linear regression, logistic regression, etc), feature engineering is an important step to improve the performance of the models. My question is does it matter if we do any feature engineering while using random forest or gradient boosting?

Granted that these models aren't deep learning models. but，it seems that some of the feature engineering methods don't really improve the model. For example: I am doing a binary classification problem, which contains about 200 features， and 20 of them are categorical features. I did the following:

1. benchmark: ran random forest classifier directly on the original data. I got AUC around 0.93, precision, recall & F-score are around 0.95 （I said around, because statifiedKfold validation is applied, and there are very small variations to the results）

2. I reduced the feature dimension by doing chi squared test and ANOVA f1 test, the run the model. results are almost identical: AUC around 0.93, precision, recall & F-score are around 0.95

3. then I one-hot keyed all the categorical features, and then rerun the model, results still almost identical: AUC around 0.93, precision, recall & F-score are around 0.95

4. Then truncated SVD is applied to reduce features further, and retrain the model, still results are unchanged...

5. at last I added polynomial term, cross term of the remaining features. results are still unchanged...

Any suggestions please? thank you.

Answer 1

It is reasonably widely recognised that feature engineering improves the outcome when using relatively advanced algorithms such as GBMs or Random Forests. There are a few reasons, relating both to overall accuracy and to useability. Firstly, if you actually want to use the model, features will require maintenance and implementation and will often require explanation to users. That is, each extra feature will create extra work. So for practical purposes, it's useful to eliminate features that don't contribute materially to improved accuracy.

With respect to overall accuracy, additional features and/or poorly engineered features increase the likelihood that you're training your model on noise rather than signal. Hence using domain knowledge or inspection of the data to suggest alternative ways to engineer features will usually improve results. The kaggle blog - blog.kaggle.com - includes 'how they did it' write-ups from podium finishers in each competition. These usually include descriptions of feature engineering - arguably more frequently than descriptions of model tuning, emphasising the importance of feature engineering - and some of them are very creative, including leveraging off domain knowledge provided by competition organisers or otherwise discovered during the competition.

This recent write-up is a good example of domain knowledge acquired during competition being used to select/ engineer features https://medium.com/kaggle-blog/2017-data-science-bowl-predicting-lung-cancer-2nd-place-solution-write-up-daniel-hammack-and-79dc345d4541 (the sections headed 'Pre-processing' and 'External Data' give good examples).

Answer 2

Yes a lot, the best way to notice this is by doing a kaggle competition. You will see that a lot of users use the same models (mostly gradient boosting and stacking) but feature engineering and selection is really what can make the difference between a top 5 percent leaderboard score and a top 20%.

But you also have to check collinearity of your features, sometimes adding too much features that are correlated can decrease the accuracy of your model.

Also you need fine tune your hyper parameters which can give you a significant boost in your model score.

After all if your model didnt improve it is likely that the algorithm you used is not suitable for your kind of problem. If an algorithm is popular it doesn't mean that it is suitable for every kind of problem.

Answer 3

Assuming that you are using trees as your booster unit (the model fitted at each iteration), it is true that some relatively simple transformations, like strictly monotonic ones (e.g. squaring, scaling, ln, etc.) will have no effect on the results, because trees are invariant to such transformations. Also, creating interactions between variables is not going to help much, because trees can inherently model interactions (provided you allow the trees to be deep enough). Check out page 352 (section 10.7) of Elements of Statistical Learning for more details.

However, I would expect the rather complex transformations that you describe, have an effect on the results in general.

Answer 4

Feature selection can make a huge difference to your test error - some features don't generalize - they improve the training error but their inclusion in the model can adversely impact the test error. I've found you can sometimes obtain a significant reduction in test loss by using a recursive process which selects a feature which has the highest rank correlation with the residual of the current best model (starting from the residuals after subtracting the mean from the targets as the initial model), add it to the model, fit and then compute the loss on a test set. If the loss goes up I drop the feature otherwise it's is added to the set of features. Repeating this process generally ends up with a better model than letting a GBRT use all available features (it is time-consuming though)