# How to fix overfitting in xgboost?

I am trying to build a classification xgboost model at work, and I'm facing overfitting issue that I have never seen before.

• My training sample size is 320,000 X 718 and testing sample is 80,000 X 78 (after doing 80-20 split)
• Features are a mix of continuous and one-hot encoded variables
• Event vs Non-Event is 50%-50% (for both training and testing)
• At the end of the day, my training accuracy is 98.07% (clearly overfitting), but my testing accuracy is also around 98.05% (testing also has 50-50% event vs non-event)

Unseen data is performing well in terms of accuracy, but its huge value seems unreal to me. I had completed the following steps for data preparation and model evaluation.:

1. replacing NULL continuous values with 0

2. removing features having correlation > 0.5 (this reduced features from some 2000+ to 718)

3. hypertuned using below parameters using 5 fold cross validation: lr = [0.01,0.05,0.1,0.2], ne = [200], md = [3,4,5]

4. important parts of my model fit:

train_X, test_X, train_y, test_y = train_test_split(X, y, test_size = 0.2, random_state=25)

xgboost = XGBClassifier(subsample = 0.8, # subsample = 0.8 ideal for big datasets
silent=False,  # whether print messages during construction
colsample_bytree = 0.4, # subsample ratio of columns when constructing each tree
gamma=10, # minimum loss reduction required to make a further partition on a leaf node of the tree, regularisation parameter
objective='binary:logistic',
eval_metric = ["error"]
)

clf = GridSearchCV(xgboost,{
'learning_rate':lr,
'n_estimators':ne,
'max_depth':md
},cv = 5,return_train_score = False)

xgboost_ht = XGBClassifier(
learning_rate = 0.2, # shrinkage for updating the rules
max_depth = 5, # maximum tree depth for base learners
n_estimators = 200, # number of boosting rounds
subsample = 0.8, # subsample = 0.8 ideal for big datasets
silent=False,  # whether print messages during construction
colsample_bytree = 0.4, # subsample ratio of columns when constructing each tree
gamma=10, # minimum loss reduction required to make a further partition on a leaf node of the tree, regularisation parameter
objective='binary:logistic',
eval_metric = ["error"]
)

xgboost_ht.fit(train_X,train_y)
y_pred = xgboost_ht.predict(test_X)
accuracy_score(y_true = test_y,y_pred = y_pred)
0.980775


I can't comprehend under what scenarios even an unseen test dataset would exhibit higher accuracy. I have normally seen test to perform lower than train, wit accuracies in the range 70%-80%.

PS - During data preparation, I had made the sample 50%-50% because originally the event proportion is 0.05%. So this is an imbalanced classification problem.

• Assuming you split the data right and there is no data leakage, then a similar performance on train and test means you've won, there is no overfitting, sometimes you luck out and can predict something very well. Nov 17, 2022 at 8:35
• I agree with @user2974951 (+1); I don't see how 98.07 vs 98.05 is indicative of over-fitting. If anything, I would try a couple of different sampling seeds to see what is the impact of sampling difference. But aside, that's very good. Nov 20, 2022 at 2:45