1
$\begingroup$

I have a feedforward ANN with a single output neuron that I use sigmoid activation on to predict true/false. I want to obtain a percentage likelihood of the true or false outcome, but when I do the appropriate reverse-engineering on the output neuron I can see that it distributes my probabilities much closer to the threshold (0.5) than I would expect and so the probability it gives me from this one output isn't very useful.

My domain space / input parameters are such that rather than there being one very obvious global minima (ie "best answer") to my problem, I suspect if one were able to visualize the multi-dimensional graph of parameters vs error that there are lots of fairly similar local minima depending on the initial seed etc. So my question is this - is it a reasonable approach to train, say, 100 models that are not perfectly overlapping in their answers (ie have all found different local minima), and use the number of models that predicted true or false as the probability of the outcome? Or is that crazy talk and is there is a well-known solution to what I'm trying to do?

Improve this question
$\endgroup$
3
  • $\begingroup$ You describe a problem of multiple local optima. The solution is to use a global optimization algorithm to locate global optimum. One simple solution is to run a local optimization from many different random starting locations and then select the best optimum. I do not know if if this is the issue that causes what you describe in first paragraph. $\endgroup$
    – Nat
    Commented May 21, 2018 at 19:53
  • $\begingroup$ My point is that the data is noisy and complex (think about modelling climate patterns etc) and there isn't an obvious global optima. So I'm trying to figure out how to use multiple local optima to generate a probability $\endgroup$
    – Matt
    Commented May 22, 2018 at 8:51
  • $\begingroup$ I understand that you want uncertainty estimates from a neural network. Perhaps this discussion is helpful eng.uber.com/neural-networks-uncertainty-estimation $\endgroup$
    – Nat
    Commented May 23, 2018 at 0:02

1 Answer 1

Reset to default
1
$\begingroup$

Training an ensemble of weak learners and throwing them in a blender is an established technique - bagging, boosting et cetera.

Just the number of predictors is, in my opinion, a pretty poor metric. If 80 models output a probability of 0.50001 and 20 of 0.499999, the actual probability if not going to be 80% here.

This seems like a good task to throw Bayes at.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.