How to fix overfitting in xgboost?

Question

I am trying to build a classification xgboost model at work, and I'm facing overfitting issue that I have never seen before.

• My training sample size is 320,000 X 718 and testing sample is 80,000 X 78 (after doing 80-20 split)
• Features are a mix of continuous and one-hot encoded variables
• Event vs Non-Event is 50%-50% (for both training and testing)
• At the end of the day, my training accuracy is 98.07% (clearly overfitting), but my testing accuracy is also around 98.05% (testing also has 50-50% event vs non-event)

Unseen data is performing well in terms of accuracy, but its huge value seems unreal to me. I had completed the following steps for data preparation and model evaluation.:

1. replacing NULL continuous values with 0

2. removing features having correlation > 0.5 (this reduced features from some 2000+ to 718)

3. hypertuned using below parameters using 5 fold cross validation: lr = [0.01,0.05,0.1,0.2], ne = [200], md = [3,4,5]

4. important parts of my model fit:

   train_X, test_X, train_y, test_y = train_test_split(X, y, test_size = 0.2, random_state=25)
   
   xgboost = XGBClassifier(subsample = 0.8, # subsample = 0.8 ideal for big datasets
                   silent=False,  # whether print messages during construction
                   colsample_bytree = 0.4, # subsample ratio of columns when constructing each tree
                   gamma=10, # minimum loss reduction required to make a further partition on a leaf node of the tree, regularisation parameter
                   objective='binary:logistic',
                   eval_metric = ["error"]
                 )
   
   clf = GridSearchCV(xgboost,{
   'learning_rate':lr,
   'n_estimators':ne,
   'max_depth':md
   },cv = 5,return_train_score = False)
   
   
   xgboost_ht = XGBClassifier(
                   learning_rate = 0.2, # shrinkage for updating the rules
                   max_depth = 5, # maximum tree depth for base learners
                   n_estimators = 200, # number of boosting rounds
                   subsample = 0.8, # subsample = 0.8 ideal for big datasets
                   silent=False,  # whether print messages during construction
                   colsample_bytree = 0.4, # subsample ratio of columns when constructing each tree
                   gamma=10, # minimum loss reduction required to make a further partition on a leaf node of the tree, regularisation parameter
                   objective='binary:logistic',
                   eval_metric = ["error"]
                 )
   
   
   xgboost_ht.fit(train_X,train_y)
   y_pred = xgboost_ht.predict(test_X)
   accuracy_score(y_true = test_y,y_pred = y_pred)
   0.980775
   

I can't comprehend under what scenarios even an unseen test dataset would exhibit higher accuracy. I have normally seen test to perform lower than train, wit accuracies in the range 70%-80%.

PS - During data preparation, I had made the sample 50%-50% because originally the event proportion is 0.05%. So this is an imbalanced classification problem.

Answer 1

To close the loop at this:

Accuracy is not a great performance metric to use (see the CV.SE thread: Why is accuracy not the best measure for assessing classification models? for an in-depth discussion) but putting that aside a difference between an Accuracy score of 98.07% in the training sest and 98.05% in the test set is not indicative of over-fitting. If anything it suggests very good alignment.

That being said: We should not up-sample/down-sample our test set. By doing that we make our test set a non-representative sample of the true population and therefore any insights about "generalisable performance" are moot. You may want to read further on this in the CV.SE thread: Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? but even if you don't read it, please do not change the class distribution in a test set (or validation set for that matter too but that is slightly less problematic). The training set is fair game for anything as ultimately most performance metrics will be upwards biased, the test set though should be as representative of "reality" as possible.