# Neural Network - Success after changing weights initialization strategy. What's the explanation?

I'm implementing a neural network in javascript to recognize handwritten digits, while studying "Neural Networks and Deep Learning" by Michael Nielsen and following the feedforward-backpropagation algorithms detailed in "A Step by Step Backpropagation Example" by Matt Mazur.

I've been struggling for the past few days to make it converge on the MNIST training samples, to no avail. My network total error always converged to 0.5 or some value around that like 0.47. No matter what I tried I could not lower the error, which made the outputs completely useless with such a high error.

I was initializing weights and biases to Math.random() (i.e. a random real number between 0 and 0.999....). After stumbling upon some questions on this site, it was mentioned that some people initialized their weights and biases to a random number between -0.5 and 0.5. I tried that and presto! My network converged to zero error in just a few training iterations. I've run it several times so this is consistently happening now.

I've also tried my algorithm with XOR and other logical functions first, and then I went for MNIST, so I tend to think my algorithm implementation is correct (although surely it can be optimized, etc).

So, how is it possible that this change in initial random values of weights made ALL the difference? Is it because I included a range of both negative and positive numbers (-0.5 to 0.5)? Should that matter? If negative weights is what the network needed, shouldn't backpropagation take care of that? (I mean, maybe taking longer to converge, but not getting stuck at 0.5 no matter how many training epochs I run).

Other info about my network just in case: it has 784 inputs, 15 hidden neurons and 10 outputs. Hidden and output network values are passed through the sigmoid function. I'm using a learning rate of 0.5.

• stats.stackexchange.com/questions/228670/…
– Sycorax
Commented Jun 20, 2017 at 21:34
• @Sycorax thanks, I've reviewed the question you mention. They are avoiding values outside a given interval (let's say keeping it between -2 and 2), keeping weights centered and truncating the normal distribution. I was setting values from 0 to 1, so my problem was not lack of truncating, but the opposite? Are you saying the same applies in this case? Commented Jun 20, 2017 at 21:45
• Have you try using smaller learning rate ? Like 0,001? Commented Jun 21, 2017 at 2:56
• @NicoCoallier yes, I tried both setting a smaller one, and a larger one, and that didn't helped. It seems it's was Sycorax suggested (sigmoid function being saturated by poorly chosen initial weights). Commented Jun 21, 2017 at 3:19