Is standard deviation of the output layer probability vector a good proxy of model confidence? I'm working on a multi-classification problem and I'm kinda new to neural networks, which is what I'm using. In the model I'm using now, the probability vector has to sum up to 1, which to me would mean little variation means little confidence. So if I'm on a 3-class classification problem and my output is:
[0.33 0.33 0.33]

There seems to be no confidence on the results, and the standard deviation is zero. On the other extreme:
[1 0 0]

There is a lot of confidence and the standard deviation is maxed out.
Is this approach okay to sort my results by confidence? Is there a better, more standard approach?
What I want to do in the end is to present all new predictions sorted by how confident the model is in its prediction. We can't deal with false positives or negatives, so each prediction has to be manually validated, and thus, it'll be better to sort said results from least confident to more confident, so the people that have to check said results can use most of their focus at the beginning.
Thanks a lot
 A: Using the variance of the probability vector is a reasonable way to capture how different the probability values are, but it may not measure model confidence in exactly the way you want. Variance over the full probability vector will look for differences in probability among even the unlikely classes, but all you really want is for one of the probabilities to be larger than all others - you probably don't care if all your unlikely classes are equally unlikely. Ideally, you'd want a measure that penalizes cases where the most likely classes are equally likely, as that indicates an inability to pick one class confidently.
Consider a probability output of {0.2, 0.3, 0.5}, which has a variance of 0.02 - this output suggests that the object is likely to be Class 3, almost twice as likely as any other class. Contrast this with a probability output of {0, 0.5, 0.5}, which has variance 0.083 - although the variance is higher, this output indicates that the object is equally likely to be Class 2 or Class 3, meaning the model has little ability to confidently predict one class versus the other. It can confidently say the object is not Class 1, but has no ability to say whether it's Class 2 or 3. As this example shows,  higher variance does not necessarily mean that the model is very "sure" about its classification - a high-variance output with equal top-ranked probabilities isn't sure of the right class at all.
Another metric you could try is the difference (or ratio) between the highest and second-highest probabilities. This will directly capture a measure of how much more likely the top class is than the next choice - a large difference in probabilities indicates the model is quite sure that the object belongs to one class and not any other. If there is little difference in the top two probabilities, the model believes either of the top two classes are equally likely, and has little confidence that the top class is actually the right one.
A: Generally the "AI/Machine Learning" people don't really view and evaluate error and probability the same way statisticians do. A "standard" way of evaluating this type of model would be with a confusion matrix, which in this case would simply be a 3X3 crosstabulation of all how each data point was classified. This is the "Machine Learning" approach to accuracy: point estimations and correct classification are more important than properly calibrated confidence.
That being said, if you're using a softmax activation function on your final layer, then by definition these are probabilities, or, perhaps more precisely, expected values across a multinomial distribution. So your interpretation is totally valid. This is a more "statistical way" of thinking about the output.
Edit: If we wanted to "sort the cases by confidence" the simplest option would be simply to sort by the variance of the vector of the three outputs. Outputs with high variance mean the model is "less confident" in its answer.
