0
$\begingroup$

I'm using a neural network mainly for binary classification. I'm using the cost function of mean squared error (cross-entropy seems to have the same results).

The problem that I'm having is that the network is either guessing 1 or 0 when optimized (class or no-class) instead of outputting the correct probability for the presence of the class.

Here's the data I'm using:

input -> output

1 -> 0 (60% of the time)

1 -> 1 (40% of the time)

2 -> 0 (40% of the time)

2 -> 1 (60% of the time)

It makes sense to just guess 1 or 0 when I sketch it out. Here is an example for when the input is zero 0--.40 would be the correct probability to output, and the neural net is correct 60% of the time:

Score-----E Correct-----E Incorrect-----Avg Error

.40--------.4----------------0.6---------------0.48

0-----------0----------------1------------------0.4

Notice that average error is much lower for guessing a 1 or 0. Is there a standard tactic for avoiding this and getting the correct probability as an output?

It seems that the same thing occurs for a) two output neurons, b) a softmax output.

How is this done to get an actual / reliable percentage output when the network is fully trained on that data?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ What is the architecture of your network (how many hidden layers, how many nodes in your hidden layers)? Also, what kind of activation functions are you using (ReLU, tanh, sigmoid)? You could be saturating the activation function by not correctly initializing weights. $\endgroup$
    – liangjy
    Jan 18, 2017 at 21:21
  • $\begingroup$ Without noise, it only needs 1 input neuron, 2 hidden, and 1 output neuron. With noise, it generally needs 30 hidden neurons (1 hidden layer) to optimize. I've tried with relu, tanh, and sigmoid. The problem is that the incentive is constant for the net to do this. My conclusion is there's never a case when a fully trained net will actually output a valid probability... $\endgroup$
    – Phyllis Sutherland
    Jan 18, 2017 at 21:26
  • $\begingroup$ Looks like I was calculating mean error and not mean squared error--this might have been responsible for issue. Looking more into it... $\endgroup$
    – Phyllis Sutherland
    Jan 18, 2017 at 22:15
  • $\begingroup$ That makes sense; the MSE when guessing 0.4 should be 0.24, while the mean error when guessing 0.4 is 0.48 (which is what you calculated above). $\endgroup$
    – liangjy
    Jan 18, 2017 at 22:32
  • $\begingroup$ Yeah that was it. All working now! $\endgroup$
    – Phyllis Sutherland
    Jan 19, 2017 at 12:02

1 Answer 1

Reset to default
1
$\begingroup$

From comments, it appears that the problem was computing mean error and not mean squared error. the MSE when guessing 0.4 should be 0.24, while the mean error when guessing 0.4 is 0.48 (which is what OP calculated).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.