Deriving confidence from multi-class prediction probabilities

Question

I have a multi-class classification model that predicts probabilities of 4 classes with the following percent of True Positives per class 0.74, 0.86, 0.87, 0.91.

I have predictions for 3 test cases:

1. (0.6, 0.1, 0.1, 0.2)  
2. (0.01, 0.09, 0.5, 0.4) 
3. (0.55, 0.25, 0.1, 0.1)

Questions:

1. Would it be correct to say that class with probability 0.6 in case 1 is a confident prediction and class with probability 0.01 in case 2 is not a confident prediction?
2. And what can we say about class with probability 0.55, is it a confident prediction or not?
3. Would it be a better way to calculate 95% confidence interval for accuracy then trying to evaluate confidence of prediction probabilities?

Thanks!

Answer 1

$1\space\&\space 2)$ A predicted probability of $0.01$ is an emphatic prediction. "I really do not expect this to happen," the model is telling you when it outputs such a prediction. On the other hand, a predicted probability of $0.6$ represents the model being fairly wishy-washy. "Maybe it will happen, but I am not particularly sure," the model is telling you.

$3)$ Accuracy has major problems (same for precision, recall, and $F_1$), one of which is that your model that outputs probabilities gets every single prediction wrong (or does it have any predicted probabilities of exactly $0$ or $1?$) and thus has $0\%$ accuracy. Calculating accuracy requires a threshold to be applied, and the linked post and links contained within explain why that is of questionable utility (and further discussion of this topic warrants a separate question, not a discussion in the comments). You might consider interval estimates for your probabilities, sure.