I'm using a neural network mainly for binary classification. I'm using the cost function of mean squared error (cross-entropy seems to have the same results).

The problem that I'm having is that the network is either guessing 1 or 0 when optimized (class or no-class) instead of outputting the correct probability for the presence of the class.

Here's the data I'm using:

input -> output

1 -> 0 (60% of the time)

1 -> 1 (40% of the time)

2 -> 0 (40% of the time)

2 -> 1 (60% of the time)

It makes sense to just guess 1 or 0 when I sketch it out. Here is an example for when the input is zero 0--.40 would be the correct probability to output, and the neural net is correct 60% of the time:

Score-----E Correct-----E Incorrect-----Avg Error



Notice that average error is much lower for guessing a 1 or 0. Is there a standard tactic for avoiding this and getting the correct probability as an output?

It seems that the same thing occurs for a) two output neurons, b) a softmax output.

How is this done to get an actual / reliable percentage output when the network is fully trained on that data?

  • 1
    $\begingroup$ What is the architecture of your network (how many hidden layers, how many nodes in your hidden layers)? Also, what kind of activation functions are you using (ReLU, tanh, sigmoid)? You could be saturating the activation function by not correctly initializing weights. $\endgroup$ – liangjy Jan 18 '17 at 21:21
  • $\begingroup$ Without noise, it only needs 1 input neuron, 2 hidden, and 1 output neuron. With noise, it generally needs 30 hidden neurons (1 hidden layer) to optimize. I've tried with relu, tanh, and sigmoid. The problem is that the incentive is constant for the net to do this. My conclusion is there's never a case when a fully trained net will actually output a valid probability... $\endgroup$ – Phyllis Sutherland Jan 18 '17 at 21:26
  • $\begingroup$ Looks like I was calculating mean error and not mean squared error--this might have been responsible for issue. Looking more into it... $\endgroup$ – Phyllis Sutherland Jan 18 '17 at 22:15
  • $\begingroup$ That makes sense; the MSE when guessing 0.4 should be 0.24, while the mean error when guessing 0.4 is 0.48 (which is what you calculated above). $\endgroup$ – liangjy Jan 18 '17 at 22:32
  • $\begingroup$ Yeah that was it. All working now! $\endgroup$ – Phyllis Sutherland Jan 19 '17 at 12:02

From comments, it appears that the problem was computing mean error and not mean squared error. the MSE when guessing 0.4 should be 0.24, while the mean error when guessing 0.4 is 0.48 (which is what OP calculated).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.