# What is the benefit of the truncated normal distribution in initializing weights in a neural network?

When initializing connection weights in a feedforward neural network, it is important to initialize them randomly to avoid any symmetries that the learning algorithm would not be able to break.

The recommendation I have seen in various places (eg. in TensorFlow's MNIST tutorial) is to use the truncated normal distribution using a standard deviation of $\dfrac{1}{\sqrt{N}}$, where $N$ is the number of inputs to the given neuron layer.

I believe that the standard deviation formula ensures that backpropagated gradients don't dissolve or amplify too quickly. But I don't know why we are using a truncated normal distribution as opposed to a regular normal distribution. Is it to avoid rare outlier weights?

• Can you provide source of this recommendation and/or the direct quotation? – Tim Aug 7 '16 at 17:54
• +Tim Good point, I added a link to an example. I believe I also saw this recommendation in a paper about neural network good practices (can't find it, though). – MiniQuark Aug 7 '16 at 18:18

I think its about saturation of the neurons. Think about you have an activation function like sigmoid.

If your weight val gets value >= 2 or <=-2 your neuron will not learn. So, if you truncate your normal distribution you will not have this issue(at least from the initialization) based on your variance. I think thats why, its better to use truncated normal in general.

• Yes, that makes sense, thanks. I think you meant "value >= 2", not 1. – MiniQuark Mar 29 '17 at 13:47
• yes it suppose to be value >= 2 – Güngör Basa Apr 8 '17 at 5:23

The benefit of using the truncated normal distribution is to prevent generating "dead neurons" due to the relu_logits being used, which is explained here.

One should generally initialize weights with a small amount of noise for symmetry breaking, and to prevent 0 gradients. Since we're using ReLU neurons, it is also good practice to initialize them with a slightly positive initial bias to avoid "dead neurons".

• I'm not sure how using the truncated_normal will prevent dead neurons: it won't add any "slightly positive initial bias". Can you please elaborate? – MiniQuark Mar 1 '17 at 15:06
• because the backpropagation will only update 'live' neurons, with some nonzero contribution to the propagation – Jason Jul 29 '18 at 10:55