Let's assume we have a network with 4 features and one neuron:

enter image description here

Accroding to the course "weight-initialization-for-deep-networks" with Andrew Ng, we want to set the variance of the weights to $Var(w)=1/n$ to avoid vanishing/exploding gradients.

My questions are:

  1. Why we want $Var(w)=1/n$ and not some other variance. Is this because the variance of the weighted sum will be 1? If $z = w_1x_1 + w_2x_2 + ... + w_nx_n$ then $Var(z)=Var(w_1x_1+...w_nx_n)=n \cdot Var(w)=n \cdot (1/n)=1$?
  2. Why we want to keep variance the same in all layers? I thought the whole idea of normaliing the weights is to make the training faster. Because if we don't normalize the weights we might get vanishing/exploding gradients. So, how keeping the variance the same will achieve this goal? I think I'm really confused

closed as unclear what you're asking by SmallChess, kjetil b halvorsen, Michael Chernick, itdxer, John Aug 2 '18 at 14:31

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ @SmallChess, I have updated my post. I hope this time it's clear what are my questions. $\endgroup$ – theateist Aug 12 '18 at 22:01

First of all, I'm pretty sure the $Var(w_1x_1+...w_nx_n)=n \cdot Var(w)$ part is incorrect, but yes in theory you are correct, as they want to keep the layer's total variance "manageable".

The goal of this initialization is to keep $Var(y_L) \approx Var(y_1)$, where $y_1$ is the output of the first layer and $y_L$ is the output of the last. In order to do so the following condition must be met, for any given layer $l$: $\frac{1}{2} n_l Var(w_l) = 1$.

This initialization strategy, was first proposed by He et al. A more detailed explanation is provided in their research paper. This strategy is most commonly referred to as He normal initialization. This technique is draws the weights from a truncated normal distribution, with $μ = 0$ and $σ = \sqrt{1 \over n_l}$, where $n_l$ is the number of neurons in the previous layer.

Another very common initialization strategy, which was the inspiration for the previous work is the Xavier (or glorot) initialization (Glorot et al.). This technique, again, draws the weights from a truncated normal distribution but this time with $μ = 0$ and $σ = \sqrt{2 \over (n_l + n_{l+1})}$, where $n_l$ is the number of neurons in the previous layer and $n_{l+1}$ is the number of neurons the current layer.


For your second question He et al, in their publication they state (pg 4, after eq 9):

A proper initialization method should avoid reducing or magnifying the magnitudes of input signals exponentially.

While they do try to keep $Var(y_L) \approx Var(y_1)$, their actual goal is to keep $Var(y_L) = c \cdot Var(y_1)$, where $c$ is a constant. At the very least they try to make the variance not grow exponentially from layer to layer.

The same motivation appears to be guiding Glorot et at, in their choice of an initializer (see Section 4.2.1)

  • $\begingroup$ @Djib20111, your answer helped me to realize what was my real confusion and question. So, I updated my post. Can you please address the second quesion? $\endgroup$ – theateist Aug 12 '18 at 22:03
  • $\begingroup$ @theateist I've edited my question as well to try to cover as best I could your second question. $\endgroup$ – Djib2011 Aug 12 '18 at 23:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.