6
$\begingroup$

Xavier initialization seems to be used quite widely now to initialize connection weights in neural networks, especially deep ones (see What are good initial weights in a neural network?).

The original paper by Xavier Glorot and Yoshua Bengio suggests initializing weights using a Uniform distribution between $-r$ and $+r$ with $r=\sqrt{\dfrac{6}{n_\text{in}+n_\text{out}}}$ (where $n_\text{in}$ and $n_\text{out}$ are the number of connections going in and out of the layer we are initializing), in order to ensure that the variance is equal to $\sigma^2 = \dfrac{2}{{n_\text{in}+n_\text{out}}}$. This helps ensure that the variance of the outputs is roughly equal to the variance of the inputs to avoid the vanishing/exploding gradients problem.

Some libraries (such as Lasagne) seem to offer the option to use the Normal distribution instead, with 0 mean and the same variance.

Is there any reason to prefer the Uniform distribution over the Normal distribution (or the reverse)? Some examples in TensorFlow's tutorials also use a truncated Normal distribution.

My guess is that the uniform distribution guarantees that no weights will be large (and so does the truncated Normal distribution). Or perhaps it just doesn't change much at all.

Any idea?

$\endgroup$
  • $\begingroup$ Studying which initialization schemes provide better performance is a hot topic right now. I'm not sure that anyone is confident that they've found the "right" solution to initializing a generic neural network, though you have identified several "good" solutions to initialize specific networks. $\endgroup$ – Sycorax Jul 4 '18 at 17:22
  • $\begingroup$ Some other papers of interest include - "[Exact solutions to the nonlinear dynamics of learning in deep linear neural networks][1]" by Andrew M. Saxe, James L. McClelland, Surya Ganguli - "[Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification][2]" by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun [1]: arxiv.org/abs/1312.6120 [2]: arxiv.org/abs/1502.01852 $\endgroup$ – Sycorax Jul 4 '18 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.