Deep Neural Network weight initialization [duplicate]

Question

Given difficult learning task (e.g high dimensionality, inherent data complexity) Deep Neural Networks become hard to train. To ease many of the problems one might:

1. Normalize && handpick quality data
2. choose a different training algorithm (e.g RMSprop instead of Gradient Descent)
3. pick a steeper gradient Cost function (e.g Cross Entropy instead of MSE)
4. Use different network structure (e.g Convolution layers instead of Feedforward)

I have heard that there are clever ways to initialize better weights. For example you can choose the magnitude better: Glorot and Bengio (2010)

• for sigmoid units: sample a Uniform(-r, r) with $r = \sqrt{\frac{6}{N_{in} + N_{out}}}$
• or hyperbolic tangent units: sample a Uniform(-r, r) with $r =4 \sqrt{\frac{6}{N_{in} + N_{out}}}$

Is there any consistent way of initializing the weights better?

Answer 1

As far as I know the two formulas you gave are pretty much the standard initialization. I did a literature review a while ago, please see my linked answer.

Answer 2

Recently, Batch Normalization was introduced for this sole purpose. Please find the paper here

Answer 3

The paper 'all you need is a good init' is a good relatively recent article about inits in deep learning. What I liked about it is that:

1. it has a short and effective literature survey on init methods, references included.
2. It achieves very good results without too many bells and whistles on cifar10.

Answer 4

Weights initialization depend on the activation function being used. Xavier and Bengio(2010) derived a method for initializing weights based on the assumption that the activations are linear. Their method resulted in the formula: \begin{align} W \sim U \left[ -\frac{\sqrt 6}{\sqrt {n_{i} + n_{i+1}}}, \frac{\sqrt 6}{\sqrt {n_{i} + n_{i+1}}} \right] \end{align}

For weights initialized using uniform distribution where $n_{i}$ represents $\text{fan in}$ and $n_{i+1}$ represents $\text{fan out}$.

He, Kaiming, et al.(2015) used a derivation method that considered use of ReLUs as the activation function and obtain a weight initialization formula:

\begin{align} W_l \sim \mathcal N \left({\Large 0}, \sqrt{\frac{2}{n_l}} \right). \end{align}

For weights initialized using Gaussian distribution whose standard deviation (std) is $\sqrt{\frac{2}{n_l}}$

Read a more comprehensive series of articles covering Mathematics behind weights initialization here.