# Why is softmax output not a good uncertainty measure for Deep Learning models?

I've been working with Convolutional Neural Networks (CNNs) for some time now, mostly on image data for semantic segmentation/instance segmentation. I've often visualized the softmax of the network output as a "heat map" to see how high per pixel activations for a certain class are. I've interpreted low activations as "uncertain" / "unconfident" and high activations as "certain" / "confident" predictions. Basically this means interpreting the softmax output (values within $(0,1)$) as a probability or (un)certainty measure of the model.

(E.g. I've interpreted an object/area with a low softmax activation averaged over its pixels to be difficult for the CNN to detect, hence the CNN being "uncertain" about predicting this kind of object.)

In my perception this often worked, and adding additional samples of "uncertain" areas to the training results improved results on these. However I've heard quite often now from different sides that using/interpreting softmax output as an (un)certainty measure is not a good idea and is generally discouraged. Why?

EDIT: To clarify what I'm asking here I'll elaborate on my insights so far in answering this question. However none of the following arguments made clear to me ** why it is generally a bad idea**, as I was repeatedly told by colleagues, supervisors and is also stated e.g. here in section "1.5"

In classification models, the probability vector obtained at the end of the pipeline (the softmax output) is often erroneously interpreted as model confidence

Although it may be tempting to interpret the values given by the final softmax layer of a convolutional neural network as confidence scores, we need to be careful not to read too much into this.

The sources above reason that using the softmax output as uncertainty measure is bad because:

imperceptible perturbations to a real image can change a deep network’s softmax output to arbitrary values

This means that softmax output isn't robust to "imperceptible perturbations" and hence it's output isn't usuable as probability.

Another paper picks up on the "softmax output = confidence" idea and argues that with this intuition networks can be easily fooled, producing "high confidence outputs for unrecognizable images".

(...) the region (in the input domain) corresponding to a particular class may be much larger than the space in that region occupied by training examples from that class. The result of this is that an image may lie within the region assigned to a class and so be classified with a large peak in the softmax output, while still being far from images that occur naturally in that class in the training set.

This means that data that is far away from training data should never get a high confidence, since the model "can't" be sure about it (as it has never seen it).

However: Isn't this generally simply questioning the generalization properties of NNs as a whole? I.e. that the NN's with softmax loss don't generalize well to (1) "imperceptible perturbations" or (2) input data samples that are far away from the training data, e.g. unrecognizable images.

Following this reasoning I still don't understand, why in practice with data that is not abstractly and artifically altered vs. the training data (i.e. most "real" applications), interpreting the softmax output as a "pseudo-probability" is a bad idea. After all, they seem to represent well what my model is sure about, even if it isn't correct (in which case I need to fix my model). And isn't model uncertainty always "only" an approximation?

• Softmax is not a uncertanity measure, it is a function that transforms values in $(-\infty, \infty)$ to $(0,1)$.
– Tim
Oct 24, 2017 at 13:08
• I have no question about what softmax is. I hope rephrasing to "why is softmax output not a good uncertainty measure" will clarify: The fact that it converts network activations to $(0,1)$ makes it tempting to interpret the network output as probability / (un)certainty measure - as I've explained. It has been made clear to me that this is not a good idea, but I still lack understanding as to why this is a bad idea. Why is a high network activation (mapped to $(0,1)$) not an indicator for high network confidence? Oct 24, 2017 at 15:39
• I like how you phrased this: "After all, they seem to represent well what my model is sure about, even if it isn't correct (in which case I need to fix my model). " Oct 17, 2019 at 23:32
• as @Tim explained the point is that softmax is outputting a probability estimate, but it doesn't give an uncertainty of it's own estimate. It's like thinking that a regression model that outputs a high value is more certain of its value, than when it outputs a low value. Jun 11, 2021 at 12:46

This question can be answered more precisely than the current answers. Fixing the deviation between the predicted probabilities (the output of the softmax layer of a neural network) and their true probabilities (which represent a notion of confidence), is known as calibration or reliability curves.

The issue with many deep neural networks is that, although they tend to perform well for prediction, their estimated predicted probabilities produced by the output of a softmax layer can not reliably be used as the true probabilities (as a confidence for each label). In practice, they tend to be too high - neural networks are 'too confident' in their predictions.

Chuan Go et. al., working with Kilian Weinberger, developed an effective solution for calibrating the predicted probabilities of neural networks in this paper: https://arxiv.org/abs/1706.04599

This paper also explains how predicted probabilities can be interpreted as confidence measures when the predicted probabilities are correctly calibrated.

• Thank you @cgnorthcutt, I've been heard people mentioning this many times and been searching for this paper ever since, really nice reference Apr 25, 2019 at 7:47
• Beautiful, thanks. Here's a link to the abstract (instead of the paper) for those interested: arxiv.org/abs/1706.04599 Sep 27, 2019 at 13:33
• "In practice, they tend to be too high - neural networks are 'too confident' in their predictions." -- what role does the use of cross entropy as the loss function play towards this effect? Jun 11, 2021 at 12:51
• you can extract confidence from logits (NOT from the softmax output) as explained here towardsdatascience.com/… , alternatively you can use MC dropout to estimate uncertainty as explained in towardsdatascience.com/monte-carlo-dropout-7fd52f8b6571 and in Aurelien Geron book "hands-on ml 2nd ed"
– alex
Jul 29 at 8:52

The relationship between softmax confidence and uncertainty is more complicated than a lot of work makes it sound.

Firstly, there are two separate issues that often get conflated.

• Callibration - Does 90% softmax confidence mean it is correct 90% of the time? This is evaluated over the training distribution. We are interested in the absolute confidence values.
• Uncertainty - Does softmax confidence reduce when the network doesn't know something? This is evaluated by comparing softmax confidence on the training distribution to some other data (often called out-of-distribution, OOD). If over the training distribution softmax confidence is in the range 92-100%, on OOD data it should be <92%. We are interested in the relative confidence values.

Callibration. Deep neural networks typically output very high softmax confidence for any input (say >95%), and are known to be poorly calibrated. As far as I know this is fairly uncontroversial. The classic reference: 'On Calibration of Modern Neural Networks' by Guo et al..

Uncertainty. This issue is less clear cut. There are well-known ways to make softmax confidence fail, such as magnifying an input, or creating adversarial examples. Softmax confidence also conflates two different sources of uncertainty (aleatoric & epistemic). These counterexamples have drawn a lot of attention, leading to claims (made with varying strength) that softmax confidence $$\neq$$ uncertainty.

What's sometimes forgotten in light of these failure modes, is that naively interpreting softmax confidence as uncertainty actually performs pretty well on many uncertainty tasks. Moreover, a lot of methods that claim to 'capture uncertainty' generally don't beat softmax confidence by all that much.

The paper, 'Understanding Softmax Confidence and Uncertainty' by Pearce et al., investigates why softmax confidence performs reasonably in these uncertainty benchmarks, describing two properties of unmodified neural networks that, in certain situations, seem to help softmax confidence $$\approx$$ uncertainty.

• very interesting. could you elaborate more on uncertainty bit? that bold "relative" refers to what? and how would softmax conflate any uncertainty? isn't it the opposite? Jun 14, 2021 at 12:41
• By relative I mean compared to the confidence over the training distribution (udpated the answer w an eg). Re: aleatoric/epistemic conflation - is the softmax saying 50% because the input looks a bit like a dog and a bit like a cat (aleatoric)? Or is it saying 50% because the input is random noise and it's never seen anything like it before (epistemic). You can't really know the reason for the uncertainty by just looking at the softmax. Sometimes this distinction matters and sometimes not. Jun 15, 2021 at 11:09

Softmax distributes the 'probability' 0-1 between the available classes. It does not express incertitude, it is not a PDF function. If you want to express the incertitude you should be looking into bayesian neural networks. Have a look at this paper: Uncertainty in Deep Learning Some rather recent probability frameworks:

Interesting keynote talk by Zoubin Ghahramani (University of Cambridge)

Have a look at this paper: Mixture Density Networks:

I guess you can implement it and add as a final layer to CONVNET. If you do implement it don't forget sharing is caring ;-) Good luck

• alternatively you can just use logits distribution/std as estimate of uncertainty and/or use MC dropout without building those complex bayesian networks, Geron's "Hands On ML" 2nd ed has a section on this.
– alex
Jul 29 at 8:56

In the paper Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, Yarin Gal and Zoubin Ghahramani argue the following

In classification, predictive probabilities obtained at the end of the pipeline (the softmax output) are often erroneously interpreted as model confidence. A model can be uncertain in its predictions even with a high softmax output (fig. 1). Passing a point estimate of a function (solid line 1a) through a softmax (solid line 1b) results in extrapolations with unjustified high confidence for points far from the training data. $$x^*$$ for example would be classified as class 1 with probability 1.

Here's figure 1.

So, if we interpret the outputs of the softmax as model uncertainty or confidence, the model is highly confident for point $$x^*$$, even though no training data was observed in that region, but this can be misleading, because the true function, in that region, could be completely different than the learned one (the solid black line).

What is called softmax in ML has the same equation as multinomial logistic equation. The latter can be used to calculate the probabilities. In practice it is widely used in estimation of default probabilities in competing risks framework for mortgages, e.g. see Eq. 4 in this paper.

Hence, I would say that your intuition is not completely off the mark. However, in the above mortgage modeling example the dependent variable is the probability metric of loan defaults. You have a pool of mortgages and observe the number of defaults. A single mortgage can either be current or in default, the probability of its default is not observable. We only observe the discrete events. However, we do model the probabilities.

How is this different from machine learning? It depends. I could decide to apply it to mortgage defaults, then it wouldn't be much different at all. On the other hand in different applications, this may not work. If you're not modeling the probability explicitly like in my example, then your model output may not represent the probability appropriately.

• Thanks for the answer. So does this boil down to: In some cases it actually would be a valid approach, however it is generally not a good idea because it isn't robust and one doesn't (easily) know whether it is applicable in a certain task? Oct 24, 2017 at 16:33
• I wouldn't be tossing around the terms like "robust" for they have very specific meaning in statistics, but you got the gist of my argument. If you are specifically modeling probabilities, you may end up using softmax equation at some point, and it would be Ok to interpret the output as a probability. In general case softmax output doesn't correspond to the probability. Oct 24, 2017 at 17:19

What is being raised is just a basic misunderstanding of statistics coming from the ML "classification" not probability viewpoint.

a predicted probability is just a prediction, if you want confidence intervals you need to do something like bootstrapping/bayesian methods.

eg if I win 6/10 games or 600/1000 my predicted estimate of winning the next game is still 60%, but the confidence interval around the 60% is much narrower after 1000 games.

• In your last example you are just using an estimation formula based on a particular prior; there are other formulae that keep into account the number of observed data. For example you could have estimates of $6/(10+2)=50\%$ vs $600/(1000+2)=59.9\%$ using a Laplace prior. Also, the point in ML classification is that we are not trying to estimate one frequency distribution, but a whole set of conditional frequency distributions – conditional on different inputs. Jun 16, 2021 at 10:53
• Sure - that's why I say:" if you want confidence intervals you need to do something like bootstrapping/bayesian methods." and then give an example... but my point is not arguing a particular methodology, just simply that the confidence is not 60% Jun 16, 2021 at 15:53