2
$\begingroup$

I am trying to build a classification xgboost model at work, and I'm facing overfitting issue that I have never seen before.

  • My training sample size is 320,000 X 718 and testing sample is 80,000 X 78 (after doing 80-20 split)
  • Features are a mix of continuous and one-hot encoded variables
  • Event vs Non-Event is 50%-50% (for both training and testing)
  • At the end of the day, my training accuracy is 98.07% (clearly overfitting), but my testing accuracy is also around 98.05% (testing also has 50-50% event vs non-event)

Unseen data is performing well in terms of accuracy, but its huge value seems unreal to me. I had completed the following steps for data preparation and model evaluation.:

  1. replacing NULL continuous values with 0

  2. removing features having correlation > 0.5 (this reduced features from some 2000+ to 718)

  3. hypertuned using below parameters using 5 fold cross validation: lr = [0.01,0.05,0.1,0.2], ne = [200], md = [3,4,5]

  4. important parts of my model fit:

    train_X, test_X, train_y, test_y = train_test_split(X, y, test_size = 0.2, random_state=25)
    
    xgboost = XGBClassifier(subsample = 0.8, # subsample = 0.8 ideal for big datasets
                    silent=False,  # whether print messages during construction
                    colsample_bytree = 0.4, # subsample ratio of columns when constructing each tree
                    gamma=10, # minimum loss reduction required to make a further partition on a leaf node of the tree, regularisation parameter
                    objective='binary:logistic',
                    eval_metric = ["error"]
                  )
    
    clf = GridSearchCV(xgboost,{
    'learning_rate':lr,
    'n_estimators':ne,
    'max_depth':md
    },cv = 5,return_train_score = False)
    
    
    xgboost_ht = XGBClassifier(
                    learning_rate = 0.2, # shrinkage for updating the rules
                    max_depth = 5, # maximum tree depth for base learners
                    n_estimators = 200, # number of boosting rounds
                    subsample = 0.8, # subsample = 0.8 ideal for big datasets
                    silent=False,  # whether print messages during construction
                    colsample_bytree = 0.4, # subsample ratio of columns when constructing each tree
                    gamma=10, # minimum loss reduction required to make a further partition on a leaf node of the tree, regularisation parameter
                    objective='binary:logistic',
                    eval_metric = ["error"]
                  )
    
    
    xgboost_ht.fit(train_X,train_y)
    y_pred = xgboost_ht.predict(test_X)
    accuracy_score(y_true = test_y,y_pred = y_pred)
    0.980775
    

I can't comprehend under what scenarios even an unseen test dataset would exhibit higher accuracy. I have normally seen test to perform lower than train, wit accuracies in the range 70%-80%.

PS - During data preparation, I had made the sample 50%-50% because originally the event proportion is 0.05%. So this is an imbalanced classification problem.

$\endgroup$
2
  • 3
    $\begingroup$ Assuming you split the data right and there is no data leakage, then a similar performance on train and test means you've won, there is no overfitting, sometimes you luck out and can predict something very well. $\endgroup$ Commented Nov 17, 2022 at 8:35
  • 2
    $\begingroup$ I agree with @user2974951 (+1); I don't see how 98.07 vs 98.05 is indicative of over-fitting. If anything, I would try a couple of different sampling seeds to see what is the impact of sampling difference. But aside, that's very good. $\endgroup$
    – usεr11852
    Commented Nov 20, 2022 at 2:45

1 Answer 1

2
$\begingroup$

To close the loop at this:

Accuracy is not a great performance metric to use (see the CV.SE thread: Why is accuracy not the best measure for assessing classification models? for an in-depth discussion) but putting that aside a difference between an Accuracy score of 98.07% in the training sest and 98.05% in the test set is not indicative of over-fitting. If anything it suggests very good alignment.

That being said: We should not up-sample/down-sample our test set. By doing that we make our test set a non-representative sample of the true population and therefore any insights about "generalisable performance" are moot. You may want to read further on this in the CV.SE thread: Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? but even if you don't read it, please do not change the class distribution in a test set (or validation set for that matter too but that is slightly less problematic). The training set is fair game for anything as ultimately most performance metrics will be upwards biased, the test set though should be as representative of "reality" as possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.