# Is it right to consider the output of the neural network as its confidence in predicting the output?

Suppose I have a single output sigmoid (tanh) that is producing an output ranging [-1, +1]. Is it right to consider this output as its confidence measue of predicting the output. The output value would be between -1 and +1 but even though I have a high accuracy I see that the values are pretty close to zero.

So, I am worried that displaying the output value * 100.0 with such low numbers might not be good. On the other hand, I am able to get a high recall and precision on the validation set.

• What loss function are you using to train the network? – Dougal Jun 22 '15 at 17:14
• I am using a mean squares error loss function – London guy Jun 22 '15 at 18:32

It is reasonable to consider the output score as a confidence measure, and this is often done. That doesn't mean, though, that you must do so if it doesn't seem to be working well, or especially that just because your network doesn't output values across the whole $[-1, 1]$ theoretical range that the network is just wishy-washy about all of its decisions.

One thing you could consider, if you're happy with the performance of the model and just want reasonable confidence scores to output, is to try to learn the probability that your model is correct given a certain output score. You can often do this by cross-validating to get many (output score, true label) pairs, and then using Platt scaling or running an isotonic regression to learn a function mapping from output scores to the probability of being positive.

• could you plz cite some paper using predictions or prediction probabilities as confidence score in case of NN? – yamini goel Jan 22 at 6:34
• @yaminigoel There are many. A paper that came out after this answer very related to this problem is arxiv.org/abs/1706.04599. – Dougal Jan 22 at 18:08
• I am not sure if this paper addresses my concern. It would be really helpful if you could answer my question here - datascience.stackexchange.com/questions/66817/… – yamini goel yesterday

In the case of a clasification feed-forward neural network, given that your output activation is a sigmoid $[0,1]$ then you actually have a binomial distribution. If you have a single output neuron you get a Bernoulli distribution is a special case of the binomial distribution with $n=1$. (https://en.wikipedia.org/wiki/Bernoulli_distribution)

Now, as you can see in wiki the variance of a prediction of a Bernoulli distribution is given by $var = p(1-p)$, so therefore you can say that the higher prediction the less variance you have, hence, the more confident you are. There is on-going research in the field as this is not a good estimate of confidence.

In case of continuous data, you could follow another approach that has been recently introduced to the field of Bayesian Optimisation, and fits your case (http://arxiv.org/abs/1502.05700). What the authors suggest is to train your network using the mean squared error of a linear output layer (after your non-linear activation tanh or sigmoid), and then train a bayesian linear regression model. With this way, you can have proper bayesian confidence intervals and it is empirically proven that they work.

More specifically, the implementation in Torch7 would be:

-- DNGO network
model = nn.Sequential()

• Thats one way to interpret the output of the network – though note that the OP actually specified tanh activations which are in $[0, 1]$. That doesn't mean assigning a Bernoulli distribution is the "right" or even a useful way to interpret it. – Dougal Jun 22 '15 at 18:50
• tanh and sigmoid activation functions have $[-1,1]$ and $[0,1]$ output spaces respectively. Therefore, if you use a sigmoid you have probabilities as outputs. – Yannis Assael Jun 22 '15 at 19:39