2
$\begingroup$

I have an XGBoost model that I fit on some X data directly out of the box:

xgb_outofbox = XGBClassifier(random_state=0).fit(X_tr, y_tr)

This model yields an F1-score of 0.626 on my test holdout data.

Then I build another model:

cv_params = {'learning_rate': [0.1, 0.2, 0.3],
             'max_depth': [2,3,4,5],
             'min_child_weight': [1,2,3,4]
             'n_estimators': [50,75,100,125]
            }
xbg_cv = GridSearchCV(XGBClassifier(random_state=0), cv_params, scoring='f1', cv=5)
xgb_cv.fit(X_tr, y_tr)

This model gives an F1-score of 0.614.

Note: the train_test_split was performed with stratify=y and the entire dataset was shuffled prior to modeling.

I have 2 questions:

  1. Why would the cross-validated model perform worse than the original when it was explicitly set to optimize F1-score?

  2. The parameters in cv_params include the exact parameters of xgb_outofbox for each item. When I call xgb_cv.best_estimator_, I get different hyperparameters than were used for the out-of-box model. Why wouldn't the cross-validated model settle on the same hyperparameters used by the out-of-box model if they yield a higher F1 score?

All other settings should be the same. I didn't specify anything differently except what I mentioned above.

$\endgroup$
1
  • 1
    $\begingroup$ 0. Welcome to CV.SE. 1. Nice question (+1), you are correct to question this behaviour. Please see my answer below for more details. $\endgroup$
    – usεr11852
    Mar 17, 2022 at 23:11

1 Answer 1

2
$\begingroup$

The cross-validated model performs worse than the "out-of-the-box" model likely because by default max_depth is 6. So when the classifier is fitted "out-of-the-box", we have more expressive base learners. In addition to that, please note that the cross-validated model is not necessarily optimal for a single hold-out test-set. It might be the case that the "out-of-the-box/depth-6" model overfits the training folds during cross-validation so it has a lower CV score than some of the less expressive models (e.g. with "depth-4") but appears better in the hold-out set.

$\endgroup$
3
  • $\begingroup$ Thanks for the response! The thing is, when I check these params for my out-of-box model, I see: learning rate: 0.1 max_depth: 3 min_child_weight: 1 n_estimaters: 100 This out-of-box model doesn't have any training folds during CV, because I don't perform CV on it. It's just fit to my training data and then used to predict on the test. I thought a model that overfit the training data would perform worse on the test data. This one isn't. It's better than a cross-validated model. I can't figure it out. $\endgroup$
    – NaiveBae
    Mar 18, 2022 at 18:50
  • 1
    $\begingroup$ OK, I am a bit surprised because as mentioned the default is 6 at the time of writing this (see xgboost.readthedocs.io/en/stable/parameter.html) but I appreciate that this is likely not what happened then. Yes, it is understood that it is fitted "once" and not via CV. As mentioned in my answer the fact that one model outperforms another model in a single hold-out set doesn't mean that is generally better, overfitting our test set can be an issue. The performance of the two models is actually quite close: $F_1$ score of 0.626 vs 0.614. Can you check the variability of the CV estimate? $\endgroup$
    – usεr11852
    Mar 18, 2022 at 21:49
  • $\begingroup$ Apologies, for the delay. Unfortunately, I can't check the variability because I have since changed everything and it would be a bit of work to go back. I'm guessing what was happening was that there was just a combination of a lucky test split and the fact that CV is training on less data with each fold validation so it arrives at slightly different hyperparameters that don't happen to perform as well on that particular test data. $\endgroup$
    – NaiveBae
    Mar 23, 2022 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.