Using an XGBoost classifier model on a few hundred thousands rows with +/- 300 numerical features and 3,000 target classes, training with multi:softproba.

Main settings for the classifier are as follow:

xgb_settings = {
    'max_depth': 12,
    'n_estimators': 160,
    'learning_rate': 0.1,
    'reg_alpha': 0.1,
    'colsample_bytree': 0.9,
    'subsample': 0.95,
    'grow_policy': 'lossguide',
    'objective': 'multi:softproba',
    'tree_method': 'gpu_hist',
    'booster': 'gbtree',

At some point around iteration 80, the training loss stops decreasing are starts growing again, while the validation accuracy still improves to the end:

training loss (green) and validation accuracy (blue)

I'm curious of why this happens and how to interpret it. It is tempting to let the training go further because the validation accuracy is improving, but is the training loss going up a problem here? Would it be a sign of overfitting or underfitting?

  • 1
    $\begingroup$ Interesting! :) Can you please describe how the validation loss is computed? Is it a single validation fold (in which case we likely overfit the validation set) or some (ideally repeated) $k$-fold schema? (which would be preferable and would suggest that we ain't overfitting "yet") Also, note that the training loss isn't "really exploding" it might be the case the training loss is at a local optimum and at this point, we are "bouncing at the bottom". Can you try a different learning rate too? Maybe this get you better results. (And maybe try AUC, Accuracy ain't a great metric.) $\endgroup$
    – usεr11852
    Commented Apr 30, 2022 at 22:17
  • $\begingroup$ Thanks for the input. Loss is categorical cross-entropy over a softmax of the predictions. You are right that this represents a single validation fold. $\endgroup$
    – Jivan
    Commented Apr 30, 2022 at 22:23
  • $\begingroup$ @usεr11852 I'm not sure I understand your idea of overfitting the validation set. Do you mean that out of pure luck, this particular validation set would be artificially favoured by the model (and that wouldn't be the case for a different fold)? I'm asking bc this is a pattern I see happening over and over with different validation sets. $\endgroup$
    – Jivan
    Commented Apr 30, 2022 at 23:06
  • 1
    $\begingroup$ (Cool, cross-entropy is fine) Primarily yes (for the over-fitting) . As mentioned, the training loss might actually be "flat" so the small up-tick we observe is not of practical significance. To state the obvious, have you tried, (aside changing the learning rates) to see if you can increase the validation set size? What is the split currently used? (Is there a chance to have a reproducible example? This phenomenon is pretty unusual.) $\endgroup$
    – usεr11852
    Commented Apr 30, 2022 at 23:24
  • $\begingroup$ @usεr11852 I've actually tried this, yes, and behaviour is similar with different combinations of validation and/or training sets — sadly I can't provide a reproducible example as it's proprietary data $\endgroup$
    – Jivan
    Commented May 1, 2022 at 13:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.