0
$\begingroup$

I'm predicting multiclass probabilities using CatBoost Classifier.

I have a balanced dataset with roughly 4000 rows, 13 features, 4 target class labels. Dataset has some outliers which I decided not to remove.

I'm using random_state=42 while splitting data and as a CatBoost parameter during both hyperparameters tuning and model evaluation with best found hyperparameters. My model training and evaluation steps:

  1. Split data into train, val, test sets in 0.7:0.15:0.15 proportion with stratification.
  2. Perform hyperparameters tuning with Optuna, using LogLoss as an evaluation metric (training on train set, evaluating on val set) and performing early stopping rounds using (X_val, y_val) as model's eval_set during hyperparameters tuning.
  3. Fit model with best found hyperparameters on train set (model.fit(X_train, y_train))
  4. Predict probabilities on X_train and X_test: model.predict_proba(X_train) and model.predict_proba(X_test)
  5. Compare metrics on train and test set, the results are following:
Log Loss AUC-ROC Brier Score ECE
Train 0.30 0.99 0.07 0.05
Test 0.55 0.94 0.08 0.02

Do these results suggest that my model is overfitting or is there something wrong with my training and evaluating steps?

The difference in LogLoss seems to be severe (which I assume is a sign of overfitting), AUC-ROC and Brier Score seem to be mostly fine (I think?), while ECE gets better on test set which I find weird in case of overfitting. Also the best Optuna trial LogLoss on validation set was nearly similar to what I got when evaluating model on test set after hyperparameters tuning.

$\endgroup$

1 Answer 1

4
$\begingroup$

TLDR: This looks like the expected amount of overfitting. IMHO it's not worth being too concerned with unless you have reason to believe it's going to cause big issues with whatever the use case for the model is (i.e. false positive could cause heavy costs).

AUC-ROC can be interpreted as $P(\hat y_1 > \hat y_0 | y_1 =1, y_0=0)$. From this definition, we can see that the metric is not very sensitive to "overconfidence" (i.e predicting $p = 0.99$ when really $p = 0.9$). Logistic loss, on the other hand, can be very sensitive to overconfidence. For example, if you predict $P(y_i = 1) = 1$ but observe $y_i = 0$, your LogLoss is unbounded!

So given the relatively small difference to AUC-ROC and moderately large difference in LogLoss, you should suspect that your model is likely to be over confident. This is pretty common for most ML methods and as long as it's not ruining model performance, you live with it. If you want to keep basically the same predictions but fix the overconfidence (i.e. nudge the predictions of $p = 0.99$ back down to $p = 0.9$), this can be done pretty easily by recalibrating your predictions. Note that recalibration will not change the rank order of predictions, but will help give you predicted probabilities you can have more confidence in; items with prediction of $p = 0.99$ should now be true 99% of the time instead of 90% if you used the uncalibrated model.

If you think the overfitting is so bad that the model is not properly leveraging the features, then you need to go back to the drawing board and think about how to improve the model (feature engineering, regularization etc).

$\endgroup$
3
  • $\begingroup$ Thank you for your thorough explanation! I tried applying sigmoid and isotonic calibration via sklearn's CalibratedClassifierCV and both calibration methods gave me worse results in all metrics than the uncalibrated model. I'm not sure how to interpret these results? I'm trying different hyperparameters tuning, and calibrators gave me better results only when I heavily decreased number of iterations which heavily decreased delta between train and test metrics, but gave worse results overall $\endgroup$
    – primadonna
    Mar 28 at 14:09
  • $\begingroup$ @primadonna I'm very surprised to hear calibration gave you worse metrics. At the very least, AUC should be unaffected, since calibration preserves the rank ordering of predictions and that's the only thing that matters for AUC. In terms of LogLoss, was this the validation error that went down? Not impossible but a little surprising. $\endgroup$
    – Cliff AB
    Mar 29 at 0:15
  • $\begingroup$ AUC was almost unaffected indeed, LogLoss went down when evaluating calibrators on test set, Brier score and ECE did as well. I posted a follow up question stats.stackexchange.com/questions/643842/… $\endgroup$
    – primadonna
    Mar 29 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.