# Brier score of calibrated probs is worse than non calibrated probs

The question is related to probability calibration and Brier score

I have faced with the following issue. I have Random forest binary classifier and then I apply isotonic regression to calibration of probabilities. The result is the following:

The question: why is Brier score of calibrated probabilities a bit worse than the one of non-calibratied probabilities? Which problem could it be?

Here is python code:

def calibrate_probas(clf, X_train, y_train, X_test, y_test, weights_test, cv):
probas = clf.predict_proba(X_test)[:, 1]

calibrator = CalibratedClassifierCV(clf, cv=cv, method='isotonic')
calibrator.fit(X_train, y_train)

calibrated_probas = calibrator.predict_proba(X_test)[:, 1]

clf_score = brier_score_loss(y_test, probas, pos_label=y_test.max(), sample_weight=weights_test)
clf_score_c = brier_score_loss(y_test, calibrated_probas, pos_label=y_test.max(), sample_weight=weights_test)

fop_c, mpv_c = calibration_curve(y_test, calibrated_probas, n_bins=10, normalize=True)
fop, mpv = calibration_curve(y_test, probas, n_bins=10, normalize=True)

# plot perfectly calibrated
f, (ax1, ax2) = plt.subplots(1, 1, figsize=(16, 6))

ax1.plot([0, 1], [0, 1], linestyle='--')

# plot model reliability
ax1.plot(mpv_c, fop_c, marker='.', label='Calibrated')
ax1.plot(mpv, fop, marker='.', c='g', label='Original')
ax1.legend()

title = f'Brier score / Brier score calib: {clf_score} / {clf_score_c}'
ax1.set_title(title)

plt.show()


Unfortunately, I can not provide the data. One of the reason, files are too big. One can see that I do not do nothing special here... Just standard python functions. Where can be the error?

• Have you decomposed the Brier score & looked at the components? Commented Nov 17, 2020 at 14:13
• Dear @gung-ReinstateMonica , could you, please, explain what you mean by "decomposing the Brier score"?
– ABK
Commented Nov 17, 2020 at 14:15
• you are calculating the brier score with weights and the graph without. What are the weights? Commented Nov 17, 2020 at 16:02
• I think @seanv507 got it. You do not use the sample weights for the calibration, but you use it to compute the two bier scores. That calibration is not the one that would minimize the bier score (as you calculate it). Commented Nov 18, 2020 at 1:59
• OK but the mismatch between your calibration curves and your brier scores is likely due to this. What happens if you generate brier score without weights (don't know if the calibration curve supports sample weights) Commented Nov 18, 2020 at 11:57

I have faced with the following issue. I have Random forest binary classifier and then I apply isotonic regression to calibration of probabilities.

The question: why is Brier score of calibrated probabilities a bit worse than the one of non-calibrated probabilities? Which problem could it be?

Whilst the graph shows that the calibrated classifier performs significantly better, the brier score loss is actually slightly better on the uncalibrated model.

Looking at the code, one sees that the brier loss is calculated with sample weights, whilst the calibration and calibration curve are calculated without sample weights.

This explains the appearance of different performance between curve and brier score.

As an aside, I would suggest not using an additional calibration step that needs its own crossvalidation, but searching for RF parameters that give good probabilities see https://arxiv.org/pdf/1812.05792.pdf

• Dear @seanv507 , could you, please, explain, why you suggest it?
– ABK
Commented Nov 18, 2020 at 22:45