2
$\begingroup$

I have a dataset with a class imbalance in favour of the positive class (85% occurence)

I'm getting a fantastically calibrated probabilities profile but balanced accuracy is 0.65 and minority recall is only 0.35 (!). Minority precision is 0.7 or so though.

I'm not really sure how to interpret this. My stakeholders are only interested in the probabilities profile but I don't know if I should trust these given that minority metrics are quite poor?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Accuracy and imbalance are irrelevant when probabilities are of interest. Smooth calibration curves are all important. $\endgroup$
    – Frank Harrell
    Commented Sep 30, 2023 at 15:42
  • $\begingroup$ It is easy to construct problems where metrics that depend on the threshold are almost arbitrarily bad even though the probabilities are calculated by the "true model" (i.e. the model used to generate the training data). See here for an example: stats.stackexchange.com/questions/539638/… where the optimal classifier assigns everything to the majority class. I agree with Frank Harrell to stay focussed on the probabilities if they are what's relevant to the application. $\endgroup$
    – Dikran Marsupial
    Commented Oct 2, 2023 at 15:29
  • $\begingroup$ Just to add, if you have a large enough dataset, class imbalance is unlikely to be a problem - if your probabilities are well calibrated, that suggests your dataset is large enough for that not to be an issue. See stats.stackexchange.com/questions/357466/… $\endgroup$
    – Dikran Marsupial
    Commented Oct 2, 2023 at 15:31

1 Answer 1

Reset to default
2
$\begingroup$

Calibration and discrimination are distinct facets of the modeling. A model can be great at one while being terrible at the other. It might be that your model has great calibration yet somewhat limited discrimination: the probability values predicted by the model are telling the truth in the sense that a predicted probability of $X$ corresponds to the event happening $X$ proportion of the time (good calibration), yet the model struggles to find differences between the two categories (discrimination).

However...

Your assessments of the probability calibration and the balanced accuracy refer to different models. The probabilities are the direct model outputs. The classifications then take those model outputs and run them through an additional step of converting the predicted probability into a categorical decision, typically by setting a threshold for classification where probability values below that threshold are categorized as class $0$, while probability values above that threshold are categorized as class $1$ (the rule for being right on the threshold is somewhat ambiguous, but it is so unlikely for a prediction to be right on the threshold value that it almost doesn't matter). This threshold is basically always $0.5$ as a software default, but that is just a software setting. Other thresholds exist and might be more reasonable, particularly in an imbalanced setting where the low prior probability of membership in the minority class often leads to a fairly low predicted ("posterior") probability of membership in the minority category, perhaps rarely or never exceeding $0.5$. This can happen even when there is fantastic calibration.

Your poor classification scores apply to just this one threshold. My suggestion is to calculate such scores across a range of thresholds. If you find a threshold or a range of thresholds where your classification metrics improve, you might find yourself feeling better about your model's discriminative performance, even if you ultimately care about the probability predictions.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.