16
$\begingroup$

When initializing connection weights in a feedforward neural network, it is important to initialize them randomly to avoid any symmetries that the learning algorithm would not be able to break.

The recommendation I have seen in various places (eg. in TensorFlow's MNIST tutorial) is to use the truncated normal distribution using a standard deviation of $\dfrac{1}{\sqrt{N}}$, where $N$ is the number of inputs to the given neuron layer.

I believe that the standard deviation formula ensures that backpropagated gradients don't dissolve or amplify too quickly. But I don't know why we are using a truncated normal distribution as opposed to a regular normal distribution. Is it to avoid rare outlier weights?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Can you provide source of this recommendation and/or the direct quotation? $\endgroup$
    – Tim ♦
    Aug 7, 2016 at 17:54
  • $\begingroup$ +Tim Good point, I added a link to an example. I believe I also saw this recommendation in a paper about neural network good practices (can't find it, though). $\endgroup$
    – MiniQuark
    Aug 7, 2016 at 18:18

2 Answers 2

Reset to default
14
$\begingroup$

I think its about saturation of the neurons. Think about you have an activation function like sigmoid.

enter image description here

If your weight val gets value >= 2 or <=-2 your neuron will not learn. So, if you truncate your normal distribution you will not have this issue(at least from the initialization) based on your variance. I think thats why, its better to use truncated normal in general.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes, that makes sense, thanks. I think you meant "value >= 2", not 1. $\endgroup$
    – MiniQuark
    Mar 29, 2017 at 13:47
  • $\begingroup$ yes it suppose to be value >= 2 $\endgroup$
    – Güngör Basa
    Apr 8, 2017 at 5:23
  • $\begingroup$ Why would my neuron not learn if the weight value was >= 2? $\endgroup$
    – Omar AlSuwaidi
    Sep 1, 2021 at 3:35
7
$\begingroup$

The truncated normal distribution is better for the parameters to be close to 0, and it's better to keep the parameters close to 0. See this question: https://stackoverflow.com/q/34569903/3552975

enter image description here

Three reasons to keep the parameters small(Source: Probabilistic Deep Learning: with Python, Keras and Tensorflow Probability):

  1. Experience shows that trained NNs often have small weights.
  2. Smaller weights lead to less extreme outputs (in classification, less extreme probabilities), which is desirable for an untrained model.
  3. It’s a known property of prediction models that adding a component to the loss function, which prefers small weights, often helps to get a higher prediction performance. This approach is also known as regularization or weight decay in non-Bayesian NNs.

And in this blog: A Gentle Introduction to Weight Constraints in Deep Learning, Dr. Jason Brownlee states that:

Smaller weights in a neural network can result in a model that is more stable and less likely to overfit the training dataset, in turn having better performance when making a prediction on new data.

If you employ ReLU you'd better make it with a slightly positive initial bias:

One should generally initialize weights with a small amount of noise for symmetry breaking, and to prevent 0 gradients. Since we're using ReLU neurons, it is also good practice to initialize them with a slightly positive initial bias to avoid "dead neurons".

Improve this answer
$\endgroup$
5
  • $\begingroup$ I'm not sure how using the truncated_normal will prevent dead neurons: it won't add any "slightly positive initial bias". Can you please elaborate? $\endgroup$
    – MiniQuark
    Mar 1, 2017 at 15:06
  • 2
    $\begingroup$ because the backpropagation will only update 'live' neurons, with some nonzero contribution to the propagation $\endgroup$
    – Jason
    Jul 29, 2018 at 10:55
  • $\begingroup$ @MiniQuark You are right. I have updated my answer. $\endgroup$
    – Lerner Zhang
    Sep 16, 2021 at 14:03
  • 1
    $\begingroup$ Can you edit your first paragraph? It seems to be missing a few words. // The quotation from the SNN paper is specifically about the self-normalizing network. It's not a general claim that all 0-mean, unit-variance activations will be propagated through the network, which they demonstrate by comparing SNNs to standard FFNs. $\endgroup$
    – Sycorax ♦
    Sep 16, 2021 at 14:09
  • $\begingroup$ @Sycorax Is it better now? Any further suggestions would be appreciated. $\endgroup$
    – Lerner Zhang
    Sep 16, 2021 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.