I'm using a neural network mainly for binary classification. I'm using the cost function of mean squared error (cross-entropy seems to have the same results).
The problem that I'm having is that the network is either guessing 1 or 0 when optimized (class or no-class) instead of outputting the correct probability for the presence of the class.
Here's the data I'm using:
input -> output
1 -> 0 (60% of the time)
1 -> 1 (40% of the time)
2 -> 0 (40% of the time)
2 -> 1 (60% of the time)
It makes sense to just guess 1 or 0 when I sketch it out. Here is an example for when the input is zero 0--.40 would be the correct probability to output, and the neural net is correct 60% of the time:
Score-----E Correct-----E Incorrect-----Avg Error
.40--------.4----------------0.6---------------0.48
0-----------0----------------1------------------0.4
Notice that average error is much lower for guessing a 1 or 0. Is there a standard tactic for avoiding this and getting the correct probability as an output?
It seems that the same thing occurs for a) two output neurons, b) a softmax output.
How is this done to get an actual / reliable percentage output when the network is fully trained on that data?