2
$\begingroup$

I already referred this post.Don't mark this as duplicate.

I am working on a binary classification problem using algos like random forest, extra trees and logistic regression. dataset shape is 977, 6. class ratio is 77:23

In terms of our metric of interest f1, random forest seemed to do better followed by extra trees and then last is logistic regression

However, in terms of calibration, I see that logistic regression is well-calibrated (not surprised), followed by extra-trees and last is random forest.

But my question, why does logistic regression have higher brier score loss when compared to random forest (which doesn't have inherent calibration capability as log reg)?

Shouldn't the logistic regression brier score loss be the smallest, followed by extra trees and last is random forest?

Please find the graphs below

enter image description here

enter image description here

enter image description here

Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ Can you add a histogram of predicted probabilities? I think the following might be the case: Logistic regression is well calibrated, but perhaps is not willing to go out on a limb in extreme cases, where as the random forest is. This would reduce the brier score for RF. $\endgroup$
    – Demetri Pananos
    Apr 2, 2022 at 18:24
  • $\begingroup$ Additionally, you should compute the loss for each prediction for each model and compare to see which observations are driving the increase in Brier score. $\endgroup$
    – Demetri Pananos
    Apr 2, 2022 at 18:27
  • $\begingroup$ @Demetri These metrics are computed on the test data and we know from the other questions about this datasets/problem that there is overfitting. So are the results surprising? $\endgroup$
    – dipetkov
    Apr 2, 2022 at 18:34
  • $\begingroup$ @dipetkov - Do you say it is overfitting based on random forest calibration curve for upper estimates? Can let me know why do you think it is overfitting? I am trying to learn because based on confusion matrix, I felt the performance between train and test is comparable. $\endgroup$
    – The Great
    Apr 2, 2022 at 18:45
  • $\begingroup$ These are your own words: "My problem is whatever I do, I see that my model overfits" taken from this post. $\endgroup$
    – dipetkov
    Apr 2, 2022 at 18:55

1 Answer 1

Reset to default
1
$\begingroup$

Brier score can be decomposed into measures of calibration and discrimination. Calibration describes the extent to which predicted probabilities align with true event occurrence. That is, if an event that is predicted to happen with probability $0.5$ actually happens $90\%$ of the time, the calibration is poor. Discrimination describes the extent to which model predictions for the two categories can be separated, and the Brier score does well here when the predicted distributions for the two categories are easy to separate (hence the relationship to the ROC AUC discussed in the link).

You have a poor Brier score despite good calibration. This must mean that the ability for a model to discriminate between the two categories is poor.

Improve this answer
$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.