I am fitting xgboost classification model to my data with highly inbalanced classes in response variable (99% vs 1%). I use cross-validation with
k=5 to tune my hyperparameters:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0) cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
and end up with the following setup:
XGBClassifier(base_score=0.5, booster=None, colsample_bylevel=1, colsample_bynode=1, colsample_bytree=0.4, disable_default_eval_metric=1, gamma=0, gpu_id=-1, importance_type='gain', interaction_constraints=None, learning_rate=0.01, max_delta_step=0, max_depth=6, min_child_weight=4, missing=nan, monotone_constraints=None, n_estimators=5000, n_jobs=0, num_parallel_tree=1, objective='binary:logistic', random_state=42, reg_alpha=0, reg_lambda=10, scale_pos_weight=1, seed=42, subsample=1, tree_method=None, validate_parameters=False, verbosity=1)
From my understanding because of how gradient boosted trees works training data will always improve with more iterations. Test score seems to stabilise around 1500 iterations. But I don't like that huge difference between training and testing errors. Would you say I am overfitting here? If so, what am I doing wrong with my hyperparameters tuning if I choose best values for each parameter and end up with model that overfits anyway? Should I try to change for example
gamma manually even despite
gamma=0 was previously selected as the best?
As a side note - despite inbalanced classes I don't use
scale_pos_weight because I mostly care about calibrated probabilites and not the exact predicted binary value.
@Edit: I have added Train/Test split and Cross Validation details. The calibration plot (on the test set) looks like below: