# Why a well-calibrated model has worse brier score loss?

I already referred this post.Don't mark this as duplicate.

I am working on a binary classification problem using algos like random forest, extra trees and logistic regression. dataset shape is 977, 6. class ratio is 77:23

In terms of our metric of interest f1, random forest seemed to do better followed by extra trees and then last is logistic regression

However, in terms of calibration, I see that logistic regression is well-calibrated (not surprised), followed by extra-trees and last is random forest.

But my question, why does logistic regression have higher brier score loss when compared to random forest (which doesn't have inherent calibration capability as log reg)?

Shouldn't the logistic regression brier score loss be the smallest, followed by extra trees and last is random forest?   • Can you add a histogram of predicted probabilities? I think the following might be the case: Logistic regression is well calibrated, but perhaps is not willing to go out on a limb in extreme cases, where as the random forest is. This would reduce the brier score for RF. Apr 2, 2022 at 18:24
• Additionally, you should compute the loss for each prediction for each model and compare to see which observations are driving the increase in Brier score. Apr 2, 2022 at 18:27
• @Demetri These metrics are computed on the test data and we know from the other questions about this datasets/problem that there is overfitting. So are the results surprising? Apr 2, 2022 at 18:34
• @dipetkov - Do you say it is overfitting based on random forest calibration curve for upper estimates? Can let me know why do you think it is overfitting? I am trying to learn because based on confusion matrix, I felt the performance between train and test is comparable. Apr 2, 2022 at 18:45
• These are your own words: "My problem is whatever I do, I see that my model overfits" taken from this post. Apr 2, 2022 at 18:55

Brier score can be decomposed into measures of calibration and discrimination. Calibration describes the extent to which predicted probabilities align with true event occurrence. That is, if an event that is predicted to happen with probability $$0.5$$ actually happens $$90\%$$ of the time, the calibration is poor. Discrimination describes the extent to which model predictions for the two categories can be separated, and the Brier score does well here when the predicted distributions for the two categories are easy to separate (hence the relationship to the ROC AUC discussed in the link).