5
$\begingroup$

Given difficult learning task (e.g high dimensionality, inherent data complexity) Deep Neural Networks become hard to train. To ease many of the problems one might:

  1. Normalize && handpick quality data
  2. choose a different training algorithm (e.g RMSprop instead of Gradient Descent)
  3. pick a steeper gradient Cost function (e.g Cross Entropy instead of MSE)
  4. Use different network structure (e.g Convolution layers instead of Feedforward)

I have heard that there are clever ways to initialize better weights. For example you can choose the magnitude better: Glorot and Bengio (2010)

  • for sigmoid units: sample a Uniform(-r, r) with $r = \sqrt{\frac{6}{N_{in} + N_{out}}}$
  • or hyperbolic tangent units: sample a Uniform(-r, r) with $r =4 \sqrt{\frac{6}{N_{in} + N_{out}}}$

Is there any consistent way of initializing the weights better?

$\endgroup$
3
$\begingroup$

As far as I know the two formulas you gave are pretty much the standard initialization. I did a literature review a while ago, please see my linked answer.

$\endgroup$
  • $\begingroup$ A potential editor suggests that the formulas for sigmoid and hyperbolic tangent be switched "to match the original paper". $\endgroup$ – gung - Reinstate Monica Aug 16 '16 at 13:48
3
$\begingroup$

The paper 'all you need is a good init' is a good relatively recent article about inits in deep learning. What I liked about it is that:

  1. it has a short and effective literature survey on init methods, references included.
  2. It achieves very good results without too many bells and whistles on cifar10.
$\endgroup$
2
$\begingroup$

Recently, Batch Normalization was introduced for this sole purpose. Please find the paper here

$\endgroup$
  • 1
    $\begingroup$ I am using this already. Is it enough by itself or maybe it can be further improved? $\endgroup$ – Joonatan Samuel Mar 28 '16 at 14:57
  • $\begingroup$ There were some extensions but i think this is the most popular one. I dont remember the exact name of the extensions. I think this should be enough also you can use a higher learning rate while optimizing. $\endgroup$ – user52705 Mar 28 '16 at 15:15
  • $\begingroup$ I have seen adaptive weight opt. algorithms work a lot better. But thanks a lot! $\endgroup$ – Joonatan Samuel Mar 28 '16 at 15:21
1
$\begingroup$

Weights initialization depend on the activation function being used. Xavier and Bengio(2010) derived a method for initializing weights based on the assumption that the activations are linear. Their method resulted in the formula: \begin{align} W \sim U \left[ -\frac{\sqrt 6}{\sqrt {n_{i} + n_{i+1}}}, \frac{\sqrt 6}{\sqrt {n_{i} + n_{i+1}}} \right] \end{align}

For weights initialized using uniform distribution where $n_{i}$ represents $\text{fan in}$ and $n_{i+1}$ represents $\text{fan out}$.

He, Kaiming, et al.(2015) used a derivation method that considered use of ReLUs as the activation function and obtain a weight initialization formula:

\begin{align} W_l \sim \mathcal N \left({\Large 0}, \sqrt{\frac{2}{n_l}} \right). \end{align}

For weights initialized using Gaussian distribution whose standard deviation (std) is $\sqrt{\frac{2}{n_l}}$

Read a more comprehensive series of articles covering Mathematics behind weights initialization here.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.