For linear models (such as linear regression, logistic regression, etc), feature engineering is an important step to improve the performance of the models. My question is does it matter if we do any feature engineering while using random forest or gradient boosting?
Granted that these models aren't deep learning models. but,it seems that some of the feature engineering methods don't really improve the model. For example: I am doing a binary classification problem, which contains about 200 features, and 20 of them are categorical features. I did the following:
benchmark: ran random forest classifier directly on the original data. I got AUC around 0.93, precision, recall & F-score are around 0.95 (I said around, because statifiedKfold validation is applied, and there are very small variations to the results)
I reduced the feature dimension by doing chi squared test and ANOVA f1 test, the run the model. results are almost identical: AUC around 0.93, precision, recall & F-score are around 0.95
then I one-hot keyed all the categorical features, and then rerun the model, results still almost identical: AUC around 0.93, precision, recall & F-score are around 0.95
Then truncated SVD is applied to reduce features further, and retrain the model, still results are unchanged...
at last I added polynomial term, cross term of the remaining features. results are still unchanged...
Any suggestions please? thank you.