Extensive hyperparameter tuning yields nothing, XGBoost classifier

Question

I'm looking for an educated reasoning concerning the following.

I have several time run extensive hyperparameter tuning sessions for an XGBoost classifier with Optuna applying large search spaces on n_estimator (100-2000), max_depth(2-14)´and gamma(1-6). In the meantime, I've had set a fixed low learning rate to 0.03 and fixed stochastic sampling (subsample, colsample_bttree and colsample_bylevel, set to 0.6, 0.6, 0.8).

However, the result doesn't improve at all compared to the default model (sklearn XGBClassifier()). Of course, the default settings perhaps are the optimal in my case. I'm using 22 features and the dataset has 200 000 observations. The dataset is imbalanced 1:20 and I'm not using scale_pos_weight.

My initial thought on this is that the dataset isn't complex enough to aloud improvements from hyperparameter tuning. However, that is just my guess. What says the experts on a situation where hyperparameter tuning doesn't yield any improvements at all for an XGBoost classifier?

Answer 1

Your datasets sounds at a superficial level reasonable large, so I would normally expect some value from hyperparameter tuning and in small datasets the right amount of regularization can be rather important. The standard settings of sklearn.XGBClassifier can of course also be reasonably decent at times, so it is possible you may sometimes not be able to do much better.

I'll assume that you already have a good evaluation set-up and are evaluating whether performance improves in an appropriate manner (e.g. via some appropriate cross-validation, where appropriate is very problem dependent see e.g. this blog).

With that caveat, I have three main candidates for what you could do differently:

1. My first suspicion is whether you are tuning the right hyperparameters. E.g. when you look at a Kaggle master’s default XGBoost tuning strategy or what the optuna LightGBMTunerCV does (yes, I know that's LightGBM, but there's massive similarities between the algorithms and what hyperparameters matter for them), you can see what they focus on. You'll notice that at least tuning subsample and colsample_bytree (and/or by level) are generally thought to be pretty important to tune. I'd focus on those parameters (note that they tune them in a fixed sequence, which can work pretty well, but you can usually do better by searching without that, it will just take much longer). You can also explore some additional ones that may help e.g. L1 or L2 regularization.

2. My first suspicion is around the n_estimator choice. Setting the learning rate to a fixed low value is usually a good approach and you certainly don't need to tune that, but you then need to make sure you use enough trees (aka estimations). I.e. make sure you look at a wide enough range for n_estimator, where values substantially higher than 1000 may be appropriate (e.g. up to 10000 or even higher). Generally, the lower you make the learning rate, the better the final model (there's drastically diminishing returns, I've never gone below 0.005) and the higher the number of trees that you need.

3. Also make sure to run enough experiments for your optuna search, but there's usually diminishing returns beyond a few thousand experiments.

Answer 2

Tunning the hyperparameters does not have to lead to improvement.

• It can be the case that the default parameters were good enough or that different values of parameters do not play an important role in training (e.g. the data is especially good or bad). Maybe you simply reached the limit of what XGBoost can achieve here.
• XGBoost is not the only machine learning model, it is still possible that other models can have better performance.
• If "no improvement" means very bad results, it always can be the case that the problem cannot be solved with machine learning.
• Check your code for bugs, if there are no changes, maybe you are doing something incorrectly. Did you make sure that you are exploring a broad enough region of the hyperparameter space?