The question is related to probability calibration and Brier score
I have faced with the following issue. I have Random forest binary classifier and then I apply isotonic regression to calibration of probabilities. The result is the following:
The question: why is Brier score of calibrated probabilities a bit worse than the one of non-calibratied probabilities? Which problem could it be?
Here is python code:
def calibrate_probas(clf, X_train, y_train, X_test, y_test, weights_test, cv):
probas = clf.predict_proba(X_test)[:, 1]
calibrator = CalibratedClassifierCV(clf, cv=cv, method='isotonic')
calibrator.fit(X_train, y_train)
calibrated_probas = calibrator.predict_proba(X_test)[:, 1]
clf_score = brier_score_loss(y_test, probas, pos_label=y_test.max(), sample_weight=weights_test)
clf_score_c = brier_score_loss(y_test, calibrated_probas, pos_label=y_test.max(), sample_weight=weights_test)
fop_c, mpv_c = calibration_curve(y_test, calibrated_probas, n_bins=10, normalize=True)
fop, mpv = calibration_curve(y_test, probas, n_bins=10, normalize=True)
# plot perfectly calibrated
f, (ax1, ax2) = plt.subplots(1, 1, figsize=(16, 6))
ax1.plot([0, 1], [0, 1], linestyle='--')
# plot model reliability
ax1.plot(mpv_c, fop_c, marker='.', label='Calibrated')
ax1.plot(mpv, fop, marker='.', c='g', label='Original')
ax1.legend()
title = f'Brier score / Brier score calib: {clf_score} / {clf_score_c}'
ax1.set_title(title)
plt.show()
Unfortunately, I can not provide the data. One of the reason, files are too big. One can see that I do not do nothing special here... Just standard python functions. Where can be the error?