3
$\begingroup$

The question is related to probability calibration and Brier score

I have faced with the following issue. I have Random forest binary classifier and then I apply isotonic regression to calibration of probabilities. The result is the following:

enter image description here

The question: why is Brier score of calibrated probabilities a bit worse than the one of non-calibratied probabilities? Which problem could it be?

Here is python code:

def calibrate_probas(clf, X_train, y_train, X_test, y_test, weights_test, cv):
    probas = clf.predict_proba(X_test)[:, 1]

    calibrator = CalibratedClassifierCV(clf, cv=cv, method='isotonic')    
    calibrator.fit(X_train, y_train)

    calibrated_probas = calibrator.predict_proba(X_test)[:, 1]

    clf_score = brier_score_loss(y_test, probas, pos_label=y_test.max(), sample_weight=weights_test)
    clf_score_c = brier_score_loss(y_test, calibrated_probas, pos_label=y_test.max(), sample_weight=weights_test)

    fop_c, mpv_c = calibration_curve(y_test, calibrated_probas, n_bins=10, normalize=True)
    fop, mpv = calibration_curve(y_test, probas, n_bins=10, normalize=True)

    # plot perfectly calibrated
    f, (ax1, ax2) = plt.subplots(1, 1, figsize=(16, 6))

    ax1.plot([0, 1], [0, 1], linestyle='--')

    # plot model reliability
    ax1.plot(mpv_c, fop_c, marker='.', label='Calibrated')
    ax1.plot(mpv, fop, marker='.', c='g', label='Original')
    ax1.legend()

    title = f'Brier score / Brier score calib: {clf_score} / {clf_score_c}'
    ax1.set_title(title)

    plt.show()

Unfortunately, I can not provide the data. One of the reason, files are too big. One can see that I do not do nothing special here... Just standard python functions. Where can be the error?

$\endgroup$
11
  • 1
    $\begingroup$ Have you decomposed the Brier score & looked at the components? $\endgroup$ Commented Nov 17, 2020 at 14:13
  • $\begingroup$ Dear @gung-ReinstateMonica , could you, please, explain what you mean by "decomposing the Brier score"? $\endgroup$
    – ABK
    Commented Nov 17, 2020 at 14:15
  • 3
    $\begingroup$ you are calculating the brier score with weights and the graph without. What are the weights? $\endgroup$
    – seanv507
    Commented Nov 17, 2020 at 16:02
  • 3
    $\begingroup$ I think @seanv507 got it. You do not use the sample weights for the calibration, but you use it to compute the two bier scores. That calibration is not the one that would minimize the bier score (as you calculate it). $\endgroup$ Commented Nov 18, 2020 at 1:59
  • 1
    $\begingroup$ OK but the mismatch between your calibration curves and your brier scores is likely due to this. What happens if you generate brier score without weights (don't know if the calibration curve supports sample weights) $\endgroup$
    – seanv507
    Commented Nov 18, 2020 at 11:57

1 Answer 1

6
+50
$\begingroup$

I have faced with the following issue. I have Random forest binary classifier and then I apply isotonic regression to calibration of probabilities.

The question: why is Brier score of calibrated probabilities a bit worse than the one of non-calibrated probabilities? Which problem could it be?

Whilst the graph shows that the calibrated classifier performs significantly better, the brier score loss is actually slightly better on the uncalibrated model.

Looking at the code, one sees that the brier loss is calculated with sample weights, whilst the calibration and calibration curve are calculated without sample weights.

This explains the appearance of different performance between curve and brier score.

As an aside, I would suggest not using an additional calibration step that needs its own crossvalidation, but searching for RF parameters that give good probabilities see https://arxiv.org/pdf/1812.05792.pdf

$\endgroup$
1
  • $\begingroup$ Dear @seanv507 , could you, please, explain, why you suggest it? $\endgroup$
    – ABK
    Commented Nov 18, 2020 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.