Is my model overfitting or is my training process wrong?

Question

I'm predicting multiclass probabilities using CatBoost Classifier.

I have a balanced dataset with roughly 4000 rows, 13 features, 4 target class labels. Dataset has some outliers which I decided not to remove.

I'm using random_state=42 while splitting data and as a CatBoost parameter during both hyperparameters tuning and model evaluation with best found hyperparameters. My model training and evaluation steps:

1. Split data into train, val, test sets in 0.7:0.15:0.15 proportion with stratification.
2. Perform hyperparameters tuning with Optuna, using LogLoss as an evaluation metric (training on train set, evaluating on val set) and performing early stopping rounds using (X_val, y_val) as model's eval_set during hyperparameters tuning.
3. Fit model with best found hyperparameters on train set (model.fit(X_train, y_train))
4. Predict probabilities on X_train and X_test: model.predict_proba(X_train) and model.predict_proba(X_test)
5. Compare metrics on train and test set, the results are following:

	Log Loss	AUC-ROC	Brier Score	ECE
Train	0.30	0.99	0.07	0.05
Test	0.55	0.94	0.08	0.02

Do these results suggest that my model is overfitting or is there something wrong with my training and evaluating steps?

The difference in LogLoss seems to be severe (which I assume is a sign of overfitting), AUC-ROC and Brier Score seem to be mostly fine (I think?), while ECE gets better on test set which I find weird in case of overfitting. Also the best Optuna trial LogLoss on validation set was nearly similar to what I got when evaluating model on test set after hyperparameters tuning.

Answer 1

TLDR: This looks like the expected amount of overfitting. IMHO it's not worth being too concerned with unless you have reason to believe it's going to cause big issues with whatever the use case for the model is (i.e. false positive could cause heavy costs).

AUC-ROC can be interpreted as $P(\hat y_1 > \hat y_0 | y_1 =1, y_0=0)$. From this definition, we can see that the metric is not very sensitive to "overconfidence" (i.e predicting $p = 0.99$ when really $p = 0.9$). Logistic loss, on the other hand, can be very sensitive to overconfidence. For example, if you predict $P(y_i = 1) = 1$ but observe $y_i = 0$, your LogLoss is unbounded!

So given the relatively small difference to AUC-ROC and moderately large difference in LogLoss, you should suspect that your model is likely to be over confident. This is pretty common for most ML methods and as long as it's not ruining model performance, you live with it. If you want to keep basically the same predictions but fix the overconfidence (i.e. nudge the predictions of $p = 0.99$ back down to $p = 0.9$), this can be done pretty easily by recalibrating your predictions. Note that recalibration will not change the rank order of predictions, but will help give you predicted probabilities you can have more confidence in; items with prediction of $p = 0.99$ should now be true 99% of the time instead of 90% if you used the uncalibrated model.

If you think the overfitting is so bad that the model is not properly leveraging the features, then you need to go back to the drawing board and think about how to improve the model (feature engineering, regularization etc).