I'm implementing a neural network in javascript to recognize handwritten digits, while studying "Neural Networks and Deep Learning" by Michael Nielsen and following the feedforward-backpropagation algorithms detailed in "A Step by Step Backpropagation Example" by Matt Mazur.
I've been struggling for the past few days to make it converge on the MNIST training samples, to no avail. My network total error always converged to 0.5 or some value around that like 0.47. No matter what I tried I could not lower the error, which made the outputs completely useless with such a high error.
I was initializing weights and biases to Math.random() (i.e. a random real number between 0 and 0.999....). After stumbling upon some questions on this site, it was mentioned that some people initialized their weights and biases to a random number between -0.5 and 0.5. I tried that and presto! My network converged to zero error in just a few training iterations. I've run it several times so this is consistently happening now.
I've also tried my algorithm with XOR and other logical functions first, and then I went for MNIST, so I tend to think my algorithm implementation is correct (although surely it can be optimized, etc).
So, how is it possible that this change in initial random values of weights made ALL the difference? Is it because I included a range of both negative and positive numbers (-0.5 to 0.5)? Should that matter? If negative weights is what the network needed, shouldn't backpropagation take care of that? (I mean, maybe taking longer to converge, but not getting stuck at 0.5 no matter how many training epochs I run).
Other info about my network just in case: it has 784 inputs, 15 hidden neurons and 10 outputs. Hidden and output network values are passed through the sigmoid function. I'm using a learning rate of 0.5.