# Why my model overfits despite selecting best hyperparameters value in each tuning step?

I am fitting xgboost classification model to my data with highly inbalanced classes in response variable (99% vs 1%). I use cross-validation with k=5 to tune my hyperparameters:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
stratify=y, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)


and end up with the following setup:

XGBClassifier(base_score=0.5, booster=None, colsample_bylevel=1,
colsample_bynode=1, colsample_bytree=0.4,
disable_default_eval_metric=1, gamma=0, gpu_id=-1,
importance_type='gain', interaction_constraints=None,
learning_rate=0.01, max_delta_step=0, max_depth=6,
min_child_weight=4, missing=nan, monotone_constraints=None,
n_estimators=5000, n_jobs=0, num_parallel_tree=1,
objective='binary:logistic', random_state=42, reg_alpha=0,
reg_lambda=10, scale_pos_weight=1, seed=42, subsample=1,
tree_method=None, validate_parameters=False, verbosity=1)


Although above values were tuned I end up with the model that I would say overfit quite heavily:

From my understanding because of how gradient boosted trees works training data will always improve with more iterations. Test score seems to stabilise around 1500 iterations. But I don't like that huge difference between training and testing errors. Would you say I am overfitting here? If so, what am I doing wrong with my hyperparameters tuning if I choose best values for each parameter and end up with model that overfits anyway? Should I try to change for example gamma manually even despite gamma=0 was previously selected as the best?

As a side note - despite inbalanced classes I don't use scale_pos_weight because I mostly care about calibrated probabilites and not the exact predicted binary value.

@Edit: I have added Train/Test split and Cross Validation details. The calibration plot (on the test set) looks like below:

• While not changing the weights and focusing on calibrated probabilities is perfectly reasonable (+1) it would make more sense to show a calibration plot to assess that. Gradient boosters usually do not give calibrated probabilities so an extra calibration step might be necessary (e.g. isotonic regression). Also, it is not mentioned how this $k$-fold cross validation has been done but it involve early stopping? How is the test/train split shown, performed? Apr 13 '20 at 23:31
• @usεr11852 I have added all the details you've asked for. Apr 14 '20 at 5:50
• Nice. Thanks. That is a rather encouraging calibration plot! :) Regarding the CV step: How did you come about using 5000 estimators? Was that supported by the CV procedure or was it fixed? That said, what you describe within the context of boosting is not unprecedented: it is widely observed that (especially in AdaBoost) the test error might decrease even after the training error is zero. See Schapire et al. (1998) "Boosting the Margin: New Explanation for the Effectiveness of Voting Methods". Apr 14 '20 at 9:56
• I have chosen 2500 in cross-validation and then noticed that there are still slight improvements aroun 2500 round so I have increased it to 5000 and set early_stopping_rounds to 250. Are you saying that my model is ok, because training error will always improve with more boosting rounds and I don't overfit? Apr 14 '20 at 14:47
• Mostly, yes. :) (I would think that 250 is a rather lenient value of early stopping rounds.) In general, over-fitting is split in two main situations: A. where both the training and test losses are decreasing, but the training loss is decreasing faster than the test loss and B. where the training loss is decreasing, but the test loss is increasing. The later (B) is clearly the problematic one for all classifiers. The former (A) is actually called optimism. Strictly speaking is bad, but not the end of the world; it is more a problem with NN cause it suggests memorisation. Apr 14 '20 at 15:24

What is shown in the learning curves presented is not unprecedented within the context of boosting. It is widely observed that (especially in AdaBoost) the test error might decrease even after the training error is zero. See Schapire et al. (1998) Boosting the Margin: New Explanation for the Effectiveness of Voting Methods for details; the general idea is that maximizing the margin can improve the generalization error of a classifier even after the training error reaches zero. By "margin" we mean the distance between the sample point and the decision boundary learned by the classifier; we usually associate it with SVM (Support Vector Machines) but it is relevant for boosting too.

Now, focusing again on the learning curve: over-fitting is split in two main situations: A. where both the training and test losses are decreasing, but the training loss is decreasing faster than the test loss and B. where the training loss is decreasing, but the test loss is increasing. The later (B) is clearly the problematic one for all classifiers. The former (A) is actually called optimism. Optimism is usually defined as the mean training error minus the mean validation error. Optimism of a model usually decreases with an increasing number of events per variable; van der Ploeg et al. (2014) Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints is an excellent and highly readable reference. Strictly speaking optimism is bad, but not the end of the world; it is more a problem with NN cause it suggests memorisation which in turn suggest issues with generalisation. That is because especially with a very large NN the capacity of it is sufficient for memorising the entire data set; Arpit et al. (2017) A Closer Look at Memorization in Deep Networks has more information on this. The same can happen with Gradient Boosting machines I suppose but I have not seen any references on the matter.

To recap, I think this model is mostly OK and does not over-fit massively, it just does not get enough "bang for its data buck" after some point! It appears to get rather optimistic after about 1000 iterations so it is worth exploring how to regularise it a bit more; for example, the subsample is set to 1 so it means we always used the whole training set when growing trees, maybe something smaller (0.80?) is more appropriate.

And a final note, the calibration plots looks good; no obvious S-shape and more or less looks monotonic. If not used already, using isotonic regression or a even a simple sigmoid on top of this classifier for some further probability calibration might help further both in terms of Brier score as well as with ranking measurements like AUC-ROC; it's no free lunch but it might help.

• I came back to that answer once again and I noticed I haven't even written a proper thank you for that comprehensive and well-defined answer. Many thanks for that! Jul 25 '20 at 9:19
• Happy to help. Thank you for your message. Jul 25 '20 at 9:44

The key here is to understand what you are trying to achieve. Any statistical model will fit its training data better than testing data and if this is unacceptable for your case all you can really do is fit a null model.

In many cases, all we care about is the testing data performance. We don't use the training performance as an indication of model performance and just ignore training performance entirely.

"Overfitting" generally refers to the point where the model is so complex that the performance on testing data is compromised (you have not reached this point with your model yet).

This would be more complicated if the training and testing data weren't entirely independent (e.g. the data were clustered, or time series data, or similar) but in the case of independent observations.

Separately, you might wish to penalise the complexity of the model (in other words, you are prepared to accept slightly poorer testing performance if the model has fewer parameters in it). This will lead to less complex models which usually means a smaller difference between training and testing performance.

• well actually even the null model (the whole sample proportion) fits slightly better in training sample than in test sample. Apr 15 '20 at 9:26
• @carlo not if you don't even let it fit the regression constant (i.e. predict 0 for everything)
– JDL
Apr 15 '20 at 9:30