I was using xgboost for binary classification task and notices that adding new features to model worsening results. As I understand xgboost itself can do implicit 'feature selection' because it tend to use more usefull features (can be seen by feature importance after model trained) and can handle feature correlation (does it really?).

So my questions is why performance is degrades when I add more features to model and what are common tips & tricks to prevent such behaviour?


Tune the hyperparameters!

Hyperparameters govern the bias-variance tradeoff of your model on that specific data set. This is explained in more detail in Elements of Statistical Learning and many other questions on this site.

XGBoost has a number of knobs to turn. A combination of hyperparameters that works well for some problem, or some feature space, might not work well for another. Different features make available different information, so what works well in one instance may not work well in another.


| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.