# Poorly calibrated probabilities but good classification in confusion matrix

I have an imbalanced data set. My goal is to balance sensitivity and specificity via the confusion matrix. I used glmnet in r with class weights. The model does well at balancing the sensitivity/specificity, but I looked at the calibration plot, and the probabilities are not well calibrated. I have read about calibrating probabilities, but I am wondering if it matters if my goal is to produce class predictions. If it does matter, I have not found a way to calibrate the probabilities when using caret::train().

• This reminds me of the situation I describe here.
– Dave
Commented Sep 18, 2022 at 22:52
• What sensitivity and specificity do you get when you don't use class weights? Much worse? Because that should give you quite well calibrated probabilities if the function of your predictors plugged into the link function is sufficiently flexible Commented Sep 19, 2022 at 0:18

This topic has been widely discussed, especially in some answers by Stephan Kolassa. I will try to summarize the main take-home messages for your specific question.

From a pure statistical point of view your interest should be on producing as output a probability for each class of any new data instance. As you deal with unbalanced data such probabilities can be small which however - as long as they are correct - is not an issue. Of course, some models can give you poor estimates of the class probabilities. In such cases, the calibration allows you to better calibrate the probabilities obtained from a given model. This means that whenever you estimate for a new observation a probability $$\hat{p}$$ of belonging to the target class, then $$\hat{p}$$ is indeed its true probability to be of that class.

If you are able to obtain a good probability estimator, then balancing sensitivity or specificity is not part of the statistical part of your problem, but rather of the decision component. Such the final decision will likely need to use some kind of threshold. Depending on the costs of type I and II errors, the cost-optimal threshold might change; however, an optimal decision might also include more than one threshold.

Ultimately, you really have to be careful about which is the specific need of the end-user of your model, because this is what is going to determine the best way of taking decisions using it.

Using class weights is pretty much guaranteed to give you badly calibrated probabilities. You end up biasing probabilities towards too low a probability for classes that you have given lower weights (or if you give higher weights to rarer classes, you are biasing the probability to be too high for those classes).

That is, unless you predictors allow you to predict (almost) perfectly. E.g. it's a whole less of an issue with say image classification, where the images make it very clear what the correct class is (so the model does not need to resort to using the prevalence of the class much in its predicted probability - although it could be a problem when the model suddenly faces e.g. very low quality images). Obviously, any scenario between a model where predictors have no value (best possible model is just to predict the class prevalence, which will be very wrong with class weights that try to "fix" those to be equal) to scenarios with lots of data + the ability to classify perfectly from the predictors.

Another option would be to correct the probabilities for the over-/under-weighting (or sampling) by adjusting the predicted logit-probability of a class with an offset that corrects for it.

Why are over-/under-sampling and class-weights used so much then? There's several reasons.

1. Some people really mostly work with scenarios where it does not do harm (possibly in combination with point 2).
2. Some models could struggle, if they see some classes too rarely. E.g. neural networks might end up getting dead neurons and become unable to predict a class, if they don't see it for too many batches in a row. This might be a genuinely good reason.
3. People focus too much on metrics defined by predicted probability </> 0.5 (e.g. accuracy defined in that way) and ignore/don't care/don't realize the problem regarding calibration.
4. Lots of software has options for it.
5. Lots of people recommend doing it, because other people have recommended doing it to them. The root cause might be (3), because as far as I'm aware there's no evidence it generally is a good idea unless one focusses on (3).