7
$\begingroup$

Suppose I have a single output sigmoid (tanh) that is producing an output ranging [-1, +1]. Is it right to consider this output as its confidence measue of predicting the output. The output value would be between -1 and +1 but even though I have a high accuracy I see that the values are pretty close to zero.

So, I am worried that displaying the output value * 100.0 with such low numbers might not be good. On the other hand, I am able to get a high recall and precision on the validation set.

|cite|improve this question
$\endgroup$
  • $\begingroup$ What loss function are you using to train the network? $\endgroup$ – Dougal Jun 22 '15 at 17:14
  • $\begingroup$ I am using a mean squares error loss function $\endgroup$ – London guy Jun 22 '15 at 18:32
6
$\begingroup$

It is reasonable to consider the output score as a confidence measure, and this is often done. That doesn't mean, though, that you must do so if it doesn't seem to be working well, or especially that just because your network doesn't output values across the whole $[-1, 1]$ theoretical range that the network is just wishy-washy about all of its decisions.

One thing you could consider, if you're happy with the performance of the model and just want reasonable confidence scores to output, is to try to learn the probability that your model is correct given a certain output score. You can often do this by cross-validating to get many (output score, true label) pairs, and then using Platt scaling or running an isotonic regression to learn a function mapping from output scores to the probability of being positive.

|cite|improve this answer
$\endgroup$
3
$\begingroup$

In the case of a clasification feed-forward neural network, given that your output activation is a sigmoid $[0,1]$ then you actually have a binomial distribution. If you have a single output neuron you get a Bernoulli distribution is a special case of the binomial distribution with $n=1$. (https://en.wikipedia.org/wiki/Bernoulli_distribution)

Now, as you can see in wiki the variance of a prediction of a Bernoulli distribution is given by $var = p(1-p)$, so therefore you can say that the higher prediction the less variance you have, hence, the more confident you are. There is on-going research in the field as this is not a good estimate of confidence.

In case of continuous data, you could follow another approach that has been recently introduced to the field of Bayesian Optimisation, and fits your case (http://arxiv.org/abs/1502.05700). What the authors suggest is to train your network using the mean squared error of a linear output layer (after your non-linear activation tanh or sigmoid), and then train a bayesian linear regression model. With this way, you can have proper bayesian confidence intervals and it is empirically proven that they work.

More specifically, the implementation in Torch7 would be:

-- DNGO network
model = nn.Sequential()
model:add(nn.Linear(ninputs,nhidden))
model:add(nn.Tanh())
model:add(nn.Linear(nhidden,nhidden))
model:add(nn.Tanh())
model:add(nn.Linear(nhidden,nhidden))
model:add(nn.Tanh())
model:add(nn.Linear(nhidden,noutputs))

Finally, the authors fit a Bayesian Linear regression to the predicted outputs against the target values, obtaining the confidence intervals. I believe this would be a very good fit for your case as well by just altering the network architecture.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thats one way to interpret the output of the network – though note that the OP actually specified tanh activations which are in $[0, 1]$. That doesn't mean assigning a Bernoulli distribution is the "right" or even a useful way to interpret it. $\endgroup$ – Dougal Jun 22 '15 at 18:50
  • $\begingroup$ tanh and sigmoid activation functions have $[-1,1]$ and $[0,1]$ output spaces respectively. Therefore, if you use a sigmoid you have probabilities as outputs. $\endgroup$ – Yannis Assael Jun 22 '15 at 19:39
  • 2
    $\begingroup$ If you use a sigmoid activation, you can interpret the output as a probability. The claim that it therefore is a probability and must be analyzed as such is, though, kind of ridiculous. When training a neural network with MSE loss, there's nothing particularly pushing the number to behave like a probability in any way, so discussing it as if it were a probability is not particularly useful. $\endgroup$ – Dougal Jun 22 '15 at 19:46
  • $\begingroup$ If you are doing classification you would train against a cross entropy criterion, where the probability would make sense. $\endgroup$ – Yannis Assael Jun 22 '15 at 22:42
  • 2
    $\begingroup$ Agreed in the case of cross-entropy objectives, though that's not what's being used here. $\endgroup$ – Dougal Jun 22 '15 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.