When initializing connection weights in a feedforward neural network, it is important to initialize them randomly to avoid any symmetries that the learning algorithm would not be able to break.

The recommendation I have seen in various places (eg. in TensorFlow's MNIST tutorial) is to use the truncated normal distribution using a standard deviation of $\dfrac{1}{\sqrt{N}}$, where $N$ is the number of inputs to the given neuron layer.

I believe that the standard deviation formula ensures that backpropagated gradients don't dissolve or amplify too quickly. But I don't know why we are using a truncated normal distribution as opposed to a regular normal distribution. Is it to avoid rare outlier weights?

  • $\begingroup$ Can you provide source of this recommendation and/or the direct quotation? $\endgroup$ – Tim Aug 7 '16 at 17:54
  • $\begingroup$ +Tim Good point, I added a link to an example. I believe I also saw this recommendation in a paper about neural network good practices (can't find it, though). $\endgroup$ – MiniQuark Aug 7 '16 at 18:18

I think its about saturation of the neurons. Think about you have an activation function like sigmoid.

enter image description here

If your weight val gets value >= 2 or <=-2 your neuron will not learn. So, if you truncate your normal distribution you will not have this issue(at least from the initialization) based on your variance. I think thats why, its better to use truncated normal in general.

  • $\begingroup$ Yes, that makes sense, thanks. I think you meant "value >= 2", not 1. $\endgroup$ – MiniQuark Mar 29 '17 at 13:47
  • $\begingroup$ yes it suppose to be value >= 2 $\endgroup$ – Güngör Basa Apr 8 '17 at 5:23

The benefit of using the truncated normal distribution is to prevent generating "dead neurons" due to the relu_logits being used, which is explained here.

One should generally initialize weights with a small amount of noise for symmetry breaking, and to prevent 0 gradients. Since we're using ReLU neurons, it is also good practice to initialize them with a slightly positive initial bias to avoid "dead neurons".

  • $\begingroup$ I'm not sure how using the truncated_normal will prevent dead neurons: it won't add any "slightly positive initial bias". Can you please elaborate? $\endgroup$ – MiniQuark Mar 1 '17 at 15:06
  • 1
    $\begingroup$ because the backpropagation will only update 'live' neurons, with some nonzero contribution to the propagation $\endgroup$ – Jason Jul 29 '18 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.