Why is softmax output not a good uncertainty measure for Deep Learning models?

Question

I've been working with Convolutional Neural Networks (CNNs) for some time now, mostly on image data for semantic segmentation/instance segmentation. I've often visualized the softmax of the network output as a "heat map" to see how high per pixel activations for a certain class are. I've interpreted low activations as "uncertain" / "unconfident" and high activations as "certain" / "confident" predictions. Basically this means interpreting the softmax output (values within $(0,1)$) as a probability or (un)certainty measure of the model.

(E.g. I've interpreted an object/area with a low softmax activation averaged over its pixels to be difficult for the CNN to detect, hence the CNN being "uncertain" about predicting this kind of object.)

In my perception this often worked, and adding additional samples of "uncertain" areas to the training results improved results on these. However I've heard quite often now from different sides that using/interpreting softmax output as an (un)certainty measure is not a good idea and is generally discouraged. Why?

---

EDIT: To clarify what I'm asking here I'll elaborate on my insights so far in answering this question. However none of the following arguments made clear to me ** why it is generally a bad idea**, as I was repeatedly told by colleagues, supervisors and is also stated e.g. here in section "1.5"

In classification models, the probability vector obtained at the end of the pipeline (the softmax output) is often erroneously interpreted as model confidence

or here in section "Background" :

Although it may be tempting to interpret the values given by the final softmax layer of a convolutional neural network as confidence scores, we need to be careful not to read too much into this.

---

The sources above reason that using the softmax output as uncertainty measure is bad because:

imperceptible perturbations to a real image can change a deep network’s softmax output to arbitrary values

This means that softmax output isn't robust to "imperceptible perturbations" and hence it's output isn't usuable as probability.

Another paper picks up on the "softmax output = confidence" idea and argues that with this intuition networks can be easily fooled, producing "high confidence outputs for unrecognizable images".

(...) the region (in the input domain) corresponding to a particular class may be much larger than the space in that region occupied by training examples from that class. The result of this is that an image may lie within the region assigned to a class and so be classified with a large peak in the softmax output, while still being far from images that occur naturally in that class in the training set.

This means that data that is far away from training data should never get a high confidence, since the model "can't" be sure about it (as it has never seen it).

However: Isn't this generally simply questioning the generalization properties of NNs as a whole? I.e. that the NN's with softmax loss don't generalize well to (1) "imperceptible perturbations" or (2) input data samples that are far away from the training data, e.g. unrecognizable images.

Following this reasoning I still don't understand, why in practice with data that is not abstractly and artifically altered vs. the training data (i.e. most "real" applications), interpreting the softmax output as a "pseudo-probability" is a bad idea. After all, they seem to represent well what my model is sure about, even if it isn't correct (in which case I need to fix my model). And isn't model uncertainty always "only" an approximation?

Answer 1

This question can be answered more precisely than the current answers. Fixing the deviation between the predicted probabilities (the output of the softmax layer of a neural network) and their true probabilities (which represent a notion of confidence), is known as calibration or reliability curves.

The issue with many deep neural networks is that, although they tend to perform well for prediction, their estimated predicted probabilities produced by the output of a softmax layer can not reliably be used as the true probabilities (as a confidence for each label). In practice, they tend to be too high - neural networks are 'too confident' in their predictions.

Chuan Guo et. al., working with Kilian Weinberger, developed an effective solution for calibrating the predicted probabilities of neural networks in this paper: On Calibration of Modern Neural Networks[1]

This paper also explains how predicted probabilities can be interpreted as confidence measures when the predicted probabilities are correctly calibrated.

[1] On Calibration of Modern Neural Networks, https://arxiv.org/abs/1706.04599

Answer 2

The relationship between softmax confidence and uncertainty is more complicated than a lot of work makes it sound.

Firstly, there are two separate issues that often get conflated.

• Callibration - Does 90% softmax confidence mean it is correct 90% of the time? This is evaluated over the training distribution. We are interested in the absolute confidence values.
• Uncertainty - Does softmax confidence reduce when the network doesn't know something? This is evaluated by comparing softmax confidence on the training distribution to some other data (often called out-of-distribution, OOD). If over the training distribution softmax confidence is in the range 92-100%, on OOD data it should be <92%. We are interested in the relative confidence values.

Callibration. Deep neural networks typically output very high softmax confidence for any input (say >95%), and are known to be poorly calibrated. As far as I know this is fairly uncontroversial. The classic reference: 'On Calibration of Modern Neural Networks' by Guo et al..

Uncertainty. This issue is less clear cut. There are well-known ways to make softmax confidence fail, such as magnifying an input, or creating adversarial examples. Softmax confidence also conflates two different sources of uncertainty (aleatoric & epistemic). These counterexamples have drawn a lot of attention, leading to claims (made with varying strength) that softmax confidence $\neq$ uncertainty.

What's sometimes forgotten in light of these failure modes, is that naively interpreting softmax confidence as uncertainty actually performs pretty well on many uncertainty tasks. Moreover, a lot of methods that claim to 'capture uncertainty' generally don't beat softmax confidence by all that much.

The paper, 'Understanding Softmax Confidence and Uncertainty' by Pearce et al., investigates why softmax confidence performs reasonably in these uncertainty benchmarks, describing two properties of unmodified neural networks that, in certain situations, seem to help softmax confidence $\approx$ uncertainty.

Answer 3

Softmax distributes the 'probability' 0-1 between the available classes. It does not express incertitude, it is not a PDF function. If you want to express the incertitude you should be looking into bayesian neural networks. Have a look at this paper: Uncertainty in Deep Learning Some rather recent probability frameworks:

• Tensorflow probability
• Edward
• Pyro pytorch

Interesting keynote talk by Zoubin Ghahramani (University of Cambridge)

Have a look at this paper: Mixture Density Networks:

I guess you can implement it and add as a final layer to CONVNET. If you do implement it don't forget sharing is caring ;-) Good luck

Answer 4

In the paper Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, Yarin Gal and Zoubin Ghahramani argue the following

In classification, predictive probabilities obtained at the end of the pipeline (the softmax output) are often erroneously interpreted as model confidence. A model can be uncertain in its predictions even with a high softmax output (fig. 1). Passing a point estimate of a function (solid line 1a) through a softmax (solid line 1b) results in extrapolations with unjustified high confidence for points far from the training data. $x^*$ for example would be classified as class 1 with probability 1.

Here's figure 1.

So, if we interpret the outputs of the softmax as model uncertainty or confidence, the model is highly confident for point $x^*$, even though no training data was observed in that region, but this can be misleading, because the true function, in that region, could be completely different than the learned one (the solid black line).

Answer 5

What is called softmax in ML has the same equation as multinomial logistic equation. The latter can be used to calculate the probabilities. In practice it is widely used in estimation of default probabilities in competing risks framework for mortgages, e.g. see Eq. 4 in this paper.

Hence, I would say that your intuition is not completely off the mark. However, in the above mortgage modeling example the dependent variable is the probability metric of loan defaults. You have a pool of mortgages and observe the number of defaults. A single mortgage can either be current or in default, the probability of its default is not observable. We only observe the discrete events. However, we do model the probabilities.

How is this different from machine learning? It depends. I could decide to apply it to mortgage defaults, then it wouldn't be much different at all. On the other hand in different applications, this may not work. If you're not modeling the probability explicitly like in my example, then your model output may not represent the probability appropriately.

Answer 6

What is being raised is just a basic misunderstanding of statistics coming from the ML "classification" not probability viewpoint.

a predicted probability is just a prediction, if you want confidence intervals you need to do something like bootstrapping/bayesian methods.

eg if I win 6/10 games or 600/1000 my predicted estimate of winning the next game is still 60%, but the confidence interval around the 60% is much narrower after 1000 games.