I'm performing multiclass probability prediction using CatBoostClassifier on a dataset with ~4000 rows, 13 features, 4 target classes. Dataset has outliers, but it is balanced.

For this task I'm using CatBoostClassifier with objective=MultiClass, eval_metric='MultiClass', num_classes=4

Here are my steps in training and evaluating the model: I'm using random_state=42 while splitting data and as a CatBoost parameter during both hyperparameters tuning and model evaluation with best found hyperparameters. My model training and evaluation steps:

  1. Split data into train, val, test sets in 0.7:0.15:0.15 proportion with stratification.
  2. Perform hyperparameters tuning with Optuna, using LogLoss as an evaluation metric (training on train set, evaluating on val set) and performing early stopping rounds using (X_val, y_val) as model's eval_set during hyperparameters tuning.
  3. Fit model with best found hyperparameters on train set (model.fit(X_train, y_train))
  4. Calibrate unfitted model using sklearn's CalibratedClassifierCV with cv=10
  5. Predict probabilities on X_train and X_test: model.predict_proba(X_train) and model.predict_proba(X_test)
  6. Compare metrics on train and test set

My results:

Calibration Train Log Loss Test Log Loss Train AUC-ROC Test AUC-ROC Train Brier Score Test Brier Score Train ECE Test ECE
Uncalibrated 0.34 0.55 0.982 0.942 0.071 0.085 0.045 0.024
Sigmoid 0.38 0.57 0.982 0.941 0.070 0.084 0.054 0.030
Isotonic 0.33 0.65 0.982 0.941 0.071 0.086 0.039 0.028

reliability_diagram probability_histogram

So from the LogLoss, Brier Score and ECE it seems like the classifier became worse overall after calibration. Is this normal?

My model is overfitting, but I was suggested that this amount of overfitting is acceptable(my task is predicting probabilities of songs belonging to 4 mood categories)

Assuming calibration is always supposed to make probability predictions better and these results are not normal, I suspect that either my model overfits too much, or my calculations of Brier Score and ECE are wrong (though it doesn't explain worse LogLoss, I used sklearn's function to calculate that)

Here are implementations of Brier score and ECE I used:

def calculate_brier_score(true_labels, predicted_probabilities):

    true_labels = np.array(true_labels)
    predicted_probabilities = np.array(predicted_probabilities)

    num_samples, num_classes = predicted_probabilities.shape

    one_hot_true = np.eye(len(predicted_probabilities[0]))[true_labels]
    probs_true = predicted_probabilities[np.arange(len(true_labels)), true_labels]

    brier_score = np.mean((probs_true - 1) ** 2)  # Squared errors for true classes
    brier_score += np.mean(predicted_probabilities ** 2, axis=1)  # Squared predicted probs for other classes
    brier_score /= len(predicted_probabilities[0])  # Normalize by number of classes

    return np.mean(brier_score)
def calculate_ece(y, proba, bins='fd'):
    # Get number of classes
    num_classes = proba.shape[1]

    ece_sum = 0.0
    for class_idx in range(num_classes):
        bin_count, bin_edges = np.histogram(proba[:, class_idx], bins=bins)
        n_bins = len(bin_count)
        bin_edges[0] -= 1e-8  # because left edge is not included
        bin_id = np.digitize(proba[:, class_idx], bin_edges, right=True) - 1
        bin_ysum = np.bincount(bin_id, weights=y == class_idx, minlength=n_bins)
        bin_probasum = np.bincount(bin_id, weights=proba[:, class_idx], minlength=n_bins)
        bin_ymean = np.divide(bin_ysum, bin_count, out=np.zeros(n_bins), where=bin_count > 0)
        bin_probamean = np.divide(bin_probasum, bin_count, out=np.zeros(n_bins), where=bin_count > 0)
        ece = np.abs((bin_probamean - bin_ymean) * bin_count).sum() / len(proba)
        ece_sum += ece

    # Calculate average ECE across all classes
    average_ece = ece_sum / num_classes
    return average_ece
  • $\begingroup$ Something is off with your Brier score calculations. You compute the mean three times and one_hot_true is never used. $\endgroup$ Commented Mar 30 at 10:24
  • 1
    $\begingroup$ Can you try varying the random state and see how much these metrics change across 100 runs? I do wonder if we're looking at difference in noise in comparing the calibrated loss vs non-calibrated. $\endgroup$
    – Cliff AB
    Commented Mar 30 at 17:08
  • $\begingroup$ @CliffAB stupid qustion, but should I change random state in both train_test_split and CatBoost parameters? Also should I perform these 100 runs after hyperparameters tuning or including the tuning? $\endgroup$
    – primadonna
    Commented Mar 30 at 20:32
  • $\begingroup$ @CliffAB I removed random_state from model parameters completely and made 100 runs of the whole procedure including hyperparameter tuning with different train_test_split random_state. 17 out of 100 runs gave better results after isotonic calibration (LogLoss, Brier Score, ECE). But sigmoid calibration gave worse (than uncalibrated) results in all 100 runs $\endgroup$
    – primadonna
    Commented Apr 2 at 21:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.