# Deriving confidence from multi-class prediction probabilities

I have a multi-class classification model that predicts probabilities of 4 classes with the following percent of True Positives per class 0.74, 0.86, 0.87, 0.91.

I have predictions for 3 test cases:

1. (0.6, 0.1, 0.1, 0.2)
2. (0.01, 0.09, 0.5, 0.4)
3. (0.55, 0.25, 0.1, 0.1)


Questions:

1. Would it be correct to say that class with probability 0.6 in case 1 is a confident prediction and class with probability 0.01 in case 2 is not a confident prediction?
2. And what can we say about class with probability 0.55, is it a confident prediction or not?
3. Would it be a better way to calculate 95% confidence interval for accuracy then trying to evaluate confidence of prediction probabilities?

Thanks!

• 1) Why don't your probabilities add up to one? $0.74 + 0.86 + 0.87 + 0.91 = 3.38$ $//$ 2) Doesn't a predicted probability of $0.01$ signal a model telling you something like, "No, absolutely not," while a predicted probability of $0.6$ signals a model telling you, "Maybe, I kind of think so, but maybe not"?
– Dave
Commented Feb 6, 2023 at 0:22
• Sorry for wrong description. 0.74, 0.86, 0.87, 0.91 are not probabilities but percent of TP in every class. Commented Feb 6, 2023 at 0:56
– Dave
Commented Feb 6, 2023 at 0:57
• You don't add percents of different classes. You add percent of TP with percent of FP to get 100 % of samples in one class. This numbers are from confusion matrix. Commented Feb 6, 2023 at 1:06

$$1\space\&\space 2)$$ A predicted probability of $$0.01$$ is an emphatic prediction. "I really do not expect this to happen," the model is telling you when it outputs such a prediction. On the other hand, a predicted probability of $$0.6$$ represents the model being fairly wishy-washy. "Maybe it will happen, but I am not particularly sure," the model is telling you.

$$3)$$ Accuracy has major problems (same for precision, recall, and $$F_1$$), one of which is that your model that outputs probabilities gets every single prediction wrong (or does it have any predicted probabilities of exactly $$0$$ or $$1?$$) and thus has $$0\%$$ accuracy. Calculating accuracy requires a threshold to be applied, and the linked post and links contained within explain why that is of questionable utility (and further discussion of this topic warrants a separate question, not a discussion in the comments). You might consider interval estimates for your probabilities, sure.

• Please clarify this: "Consider what the predicted probabilities would be if you flipped the categorical encoding (zero to one, one to zero). Your old predicted P(Y=1)=0.01 would then be P(Y=1)=0.99 , while your old P(Y=1)=0.60 and P(Y=1)=0.55 would then be P(Y=1)=0.40 and P(Y=1)=0.45 , respectively. What do you mean by ' flipped the categorical encoding ' ? And also, I have 4 classes, not just 2. thanks Commented Feb 6, 2023 at 1:26
• @dokondr I have removed that discussion, since it does not apply (at least not in an obvious way) to a situation with four categories.
– Dave
Commented Feb 6, 2023 at 1:40
• I see what you mean, thanks. On another point: Confidence interval may be calculated not only for accuracy, but for other statistics as well: P, R and F1, it should be more informative, I think. Commented Feb 6, 2023 at 1:40
• @dokondr This discussion warrants its own question (and there are plenty), but all of those statistics suffer from problems similar to that of accuracy. Please refer to the linked question and the links within.
– Dave
Commented Feb 6, 2023 at 1:42
• Thanks, I will. It still will be great if you could give a short answer how to estimate confidence of class prediction in multi-class model. This short answer would work as a course for me to further investigate on my own, without asking more questions here. This was a whole point of my initial question here. Commented Feb 6, 2023 at 1:51