18
$\begingroup$

I've been working with Convolutional Neural Networks (CNNs) for some time now, mostly on image data for semantic segmentation/instance segmentation. I've often visualized the softmax of the network output as a "heat map" to see how high per pixel activations for a certain class are. I've interpreted low activations as "uncertain" / "unconfident" and high activations as "certain" / "confident" predictions. Basically this means interpreting the softmax output (values within $(0,1)$) as a probability or (un)certainty measure of the model.

(E.g. I've interpreted an object/area with a low softmax activation averaged over its pixels to be difficult for the CNN to detect, hence the CNN being "uncertain" about predicting this kind of object.)

In my perception this often worked, and adding additional samples of "uncertain" areas to the training results improved results on these. However I've heard quite often now from different sides that using/interpreting softmax output as an (un)certainty measure is not a good idea and is generally discouraged. Why?


EDIT: To clarify what I'm asking here I'll elaborate on my insights so far in answering this question. However none of the following arguments made clear to me ** why it is generally a bad idea**, as I was repeatedly told by colleagues, supervisors and is also stated e.g. here in section "1.5"

In classification models, the probability vector obtained at the end of the pipeline (the softmax output) is often erroneously interpreted as model confidence

or here in section "Background" :

Although it may be tempting to interpret the values given by the final softmax layer of a convolutional neural network as confidence scores, we need to be careful not to read too much into this.


The sources above reason that using the softmax output as uncertainty measure is bad because:

imperceptible perturbations to a real image can change a deep network’s softmax output to arbitrary values

This means that softmax output isn't robust to "imperceptible perturbations" and hence it's output isn't usuable as probability.

Another paper picks up on the "softmax output = confidence" idea and argues that with this intuition networks can be easily fooled, producing "high confidence outputs for unrecognizable images".

(...) the region (in the input domain) corresponding to a particular class may be much larger than the space in that region occupied by training examples from that class. The result of this is that an image may lie within the region assigned to a class and so be classified with a large peak in the softmax output, while still being far from images that occur naturally in that class in the training set.

This means that data that is far away from training data should never get a high confidence, since the model "can't" be sure about it (as it has never seen it).

However: Isn't this generally simply questioning the generalization properties of NNs as a whole? I.e. that the NN's with softmax loss don't generalize well to (1) "imperceptible perturbations" or (2) input data samples that are far away from the training data, e.g. unrecognizable images.

Following this reasoning I still don't understand, why in practice with data that is not abstractly and artifically altered vs. the training data (i.e. most "real" applications), interpreting the softmax output as a "pseudo-probability" is a bad idea. After all, they seem to represent well what my model is sure about, even if it isn't correct (in which case I need to fix my model). And isn't model uncertainty always "only" an approximation?

$\endgroup$
  • 3
    $\begingroup$ Softmax is not a uncertanity measure, it is a function that transforms values in $(-\infty, \infty)$ to $(0,1)$. $\endgroup$ – Tim Oct 24 '17 at 13:08
  • 1
    $\begingroup$ I have no question about what softmax is. I hope rephrasing to "why is softmax output not a good uncertainty measure" will clarify: The fact that it converts network activations to $(0,1)$ makes it tempting to interpret the network output as probability / (un)certainty measure - as I've explained. It has been made clear to me that this is not a good idea, but I still lack understanding as to why this is a bad idea. Why is a high network activation (mapped to $(0,1)$) not an indicator for high network confidence? $\endgroup$ – Honeybear Oct 24 '17 at 15:39
11
$\begingroup$

This question can be answered more precisely than the current answers. Fixing the deviation between the predicted probabilities (the output of the softmax layer of a neural network) and their true probabilities (which represent a notion of confidence), is known as calibration or reliability curves.

The issue with many deep neural networks is that, although they tend to perform well for prediction, their estimated predicted probabilities produced by the output of a softmax layer can not reliably be used as the true probabilities (as a confidence for each label). In practice, they tend to be too high - neural networks are 'too confident' in their predictions.

Chuan Go et. al., working with Kilian Weinberger, developed an effective solution for calibrating the predicted probabilities of neural networks in this paper: https://arxiv.org/pdf/1706.04599.pdf

This paper also explains how predicted probabilities can be interpreted as confidence measures when the predicted probabilities are correctly calibrated.

$\endgroup$
  • 1
    $\begingroup$ Thank you @cgnorthcutt, I've been heard people mentioning this many times and been searching for this paper ever since, really nice reference $\endgroup$ – Théophile Pace Apr 25 at 7:47
  • $\begingroup$ Beautiful, thanks. Here's a link to the abstract (instead of the paper) for those interested: arxiv.org/abs/1706.04599 $\endgroup$ – Waylon Flinn Sep 27 at 13:33
3
$\begingroup$

What is called softmax in ML has the same equation as multinomial logistic equation. The latter can be used to calculate the probabilities. In practice it is widely used in estimation of default probabilities in competing risks framework for mortgages, e.g. see Eq. 4 in this paper.

Hence, I would say that your intuition is not completely off the mark. However, in the above mortgage modeling example the dependent variable is the probability metric of loan defaults. You have a pool of mortgages and observe the number of defaults. A single mortgage can either be current or in default, the probability of its default is not observable. We only observe the discrete events. However, we do model the probabilities.

How is this different from machine learning? It depends. I could decide to apply it to mortgage defaults, then it wouldn't be much different at all. On the other hand in different applications, this may not work. If you're not modeling the probability explicitly like in my example, then your model output may not represent the probability appropriately.

$\endgroup$
  • $\begingroup$ Thanks for the answer. So does this boil down to: In some cases it actually would be a valid approach, however it is generally not a good idea because it isn't robust and one doesn't (easily) know whether it is applicable in a certain task? $\endgroup$ – Honeybear Oct 24 '17 at 16:33
  • $\begingroup$ I wouldn't be tossing around the terms like "robust" for they have very specific meaning in statistics, but you got the gist of my argument. If you are specifically modeling probabilities, you may end up using softmax equation at some point, and it would be Ok to interpret the output as a probability. In general case softmax output doesn't correspond to the probability. $\endgroup$ – Aksakal Oct 24 '17 at 17:19
1
$\begingroup$

Softmax distributes the 'probability' 0-1 between the available classes. It does not express incertitude, it is not a PDF function. If you want to express the incertitude you should be looking into bayesian neural networks. Have a look at this paper: Uncertainty in Deep Learning Some rather recent probability frameworks:

Interesting keynote talk by Zoubin Ghahramani (University of Cambridge)

Have a look at this paper: Mixture Density Networks:

I guess you can implement it and add as a final layer to CONVNET. If you do implement it don't forget sharing is caring ;-) Good luck

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.