Deep Neural Network weight initialization Given difficult learning task (e.g high dimensionality, inherent data complexity) Deep Neural Networks become hard to train.
 To ease many of the problems one might:


*

*Normalize && handpick quality data 

*choose a different training algorithm (e.g RMSprop instead of Gradient Descent)

*pick a steeper gradient Cost function (e.g Cross Entropy instead of MSE)

*Use different network structure (e.g Convolution layers instead of Feedforward)


I have heard that there are clever ways to initialize better weights. For example you can choose the magnitude better:
Glorot and Bengio (2010)


*

*for sigmoid units: sample a Uniform(-r, r) with $r =
   \sqrt{\frac{6}{N_{in} + N_{out}}}$

*or hyperbolic tangent units: sample a Uniform(-r, r) with $r =4
   \sqrt{\frac{6}{N_{in} + N_{out}}}$


Is there any consistent way of initializing the weights better?
 A: Recently, Batch Normalization was introduced for this sole purpose. Please find the paper here
A: The paper 'all you need is a good init' is a good relatively recent article about inits in deep learning. What I liked about it is that:


*

*it has a short and effective literature survey on init methods, references included.

*It achieves very good results without too many bells and whistles on cifar10.

A: As far as I know the two formulas you gave are pretty much the standard initialization. I did a literature review a while ago, please see my linked answer.
A: Weights initialization depend on the activation function being used. Xavier and Bengio(2010) derived a method for initializing weights based on the
assumption that the activations are linear. Their method resulted in the formula:
\begin{align}
W \sim U \left[ -\frac{\sqrt 6}{\sqrt {n_{i} + n_{i+1}}}, \frac{\sqrt 6}{\sqrt {n_{i} + n_{i+1}}} \right] 
\end{align}
For weights initialized using uniform distribution where $n_{i}$ represents $\text{fan in}$ and  $n_{i+1}$ represents $\text{fan out}$.
He, Kaiming, et al.(2015) used a derivation method that considered use of ReLUs as the activation function and obtain a weight initialization formula:
\begin{align}
W_l \sim \mathcal N \left({\Large 0}, \sqrt{\frac{2}{n_l}} \right).
\end{align}
For weights initialized using Gaussian distribution whose standard deviation (std) is $\sqrt{\frac{2}{n_l}}$
Read a more comprehensive series of articles covering Mathematics behind weights initialization here.
