8
$\begingroup$

In the week 5 lecture notes for Andrew Ng's Coursera Machine Learning Class, the following formula is given for calculating the value of $\epsilon$ used to initialise $\Theta$ with random values:

Forumla for calculating epsilon-init for random initialisation

In the exercise, further clarification is given:

One effective strategy for choosing $\epsilon_{init}$ is to base it on the number of units in the network. A good choice of $\epsilon_{init}$ is $\epsilon_{init} = \frac{\sqrt{6}}{\sqrt{L_{in} - L_{out}}}$, where $L_{in} = s_l$ and $L_{out} = s_{l+1}$ are the number of units in the layers adjacent to $\Theta^{(l)}$.

Why is the constant $\sqrt 6$ used here? Why not $\sqrt 5$, $\sqrt 7$ or $\sqrt {6.1}$?

|cite|improve this question
$\endgroup$
4
$\begingroup$

I believe this is Xavier normalized initialization (implemented in several deep learning frameworks eg Keras, Cafe, ...) from Understanding the difficulty of training deep feedforward neural networks by Xavier Glorot & Yoshua Bengio.

See equations 12, 15 and 16 in the paper linked: they aim to satisfy equation 12: $$\text{Var}[W_i] = \frac{2}{n_i + n_{i+1}}$$

and the variance of a uniform RV in $[-\epsilon,\epsilon]$ is $\epsilon^2/3$ (mean is zero, pdf = $1/(2\epsilon)$ so variance $=\int_{-\epsilon}^{\epsilon}x^2 \frac{1}{2\epsilon}dx$

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Hmm, so why use $\sqrt 6$ instead of $2$? $\endgroup$ – Tom Hale Jul 18 '17 at 6:28
  • $\begingroup$ Plug epsilon into the formula for variance of uniform random variable in +/-x and what do you get? $\endgroup$ – seanv507 Jul 18 '17 at 18:26
  • $\begingroup$ Doh! I now see in formulae (16) that $[-\epsilon, \epsilon]$ is used. Where do you get $x^2 / 3$ from though? $\endgroup$ – Tom Hale Jul 23 '17 at 10:45
  • 1
    $\begingroup$ added explanation for variance of uniform RV... $\endgroup$ – seanv507 Jul 23 '17 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.