I've been working with Convolutional Neural Networks (CNNs) for some time now, mostly on image data for semantic segmentation/instance segmentation. I've often visualized the softmax of the network output as a "heat map" to see how high per pixel activations for a certain class are. I've interpreted low activations as "uncertain" / "unconfident" and high activations as "certain" / "confident" predictions. Basically this means interpreting the softmax output (values within $(0,1)$) as a probability or (un)certainty measure of the model.
(E.g. I've interpreted an object/area with a low softmax activation averaged over its pixels to be difficult for the CNN to detect, hence the CNN being "uncertain" about predicting this kind of object.)
In my perception this often worked, and adding additional samples of "uncertain" areas to the training results improved results on these. However I've heard quite often now from different sides that using/interpreting softmax output as an (un)certainty measure is not a good idea and is generally discouraged. Why?
EDIT: To clarify what I'm asking here I'll elaborate on my insights so far in answering this question. However none of the following arguments made clear to me ** why it is generally a bad idea**, as I was repeatedly told by colleagues, supervisors and is also stated e.g. here in section "1.5"
In classification models, the probability vector obtained at the end of the pipeline (the softmax output) is often erroneously interpreted as model confidence
Although it may be tempting to interpret the values given by the final softmax layer of a convolutional neural network as confidence scores, we need to be careful not to read too much into this.
The sources above reason that using the softmax output as uncertainty measure is bad because:
imperceptible perturbations to a real image can change a deep network’s softmax output to arbitrary values
This means that softmax output isn't robust to "imperceptible perturbations" and hence it's output isn't usuable as probability.
Another paper picks up on the "softmax output = confidence" idea and argues that with this intuition networks can be easily fooled, producing "high confidence outputs for unrecognizable images".
(...) the region (in the input domain) corresponding to a particular class may be much larger than the space in that region occupied by training examples from that class. The result of this is that an image may lie within the region assigned to a class and so be classified with a large peak in the softmax output, while still being far from images that occur naturally in that class in the training set.
This means that data that is far away from training data should never get a high confidence, since the model "can't" be sure about it (as it has never seen it).
However: Isn't this generally simply questioning the generalization properties of NNs as a whole? I.e. that the NN's with softmax loss don't generalize well to (1) "imperceptible perturbations" or (2) input data samples that are far away from the training data, e.g. unrecognizable images.
Following this reasoning I still don't understand, why in practice with data that is not abstractly and artifically altered vs. the training data (i.e. most "real" applications), interpreting the softmax output as a "pseudo-probability" is a bad idea. After all, they seem to represent well what my model is sure about, even if it isn't correct (in which case I need to fix my model). And isn't model uncertainty always "only" an approximation?