# What are good initial weights in a neural network?

I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. It is assumed, that the sets are normalized - mean 0, variance 1 (don't know if this matters).

Why is this a good idea?

-

I assume you are using logistic neurons, and that you are training by gradient descent/back-propagation.

The logistic function is close to flat for large positive or negative inputs. The derivative at an input of $2$ is about $1/10$, but at $10$ the derivative is about $1/22000$ . This means that if the input of a logistic neuron is $10$ then, for a given training signal, the neuron will learn about $2200$ times slower that if the input was $2$.

If you want the neuron to learn quickly, you either need to produce a huge training signal (such as with a cross-entropy loss function) or you want the derivative to be large. To make the derivative large, you set the initial weights so that you often get inputs in the range $[-4,4]$.

The initial weights you give might or might not work. It depends on how the inputs are normalized. If the inputs are normalized to have mean $0$ and standard deviation $1$, then a random sum of $d$ terms with weights uniform on $(\frac{-1}{\sqrt{d}},\frac{1}{\sqrt{d}})$ will have mean $0$ and variance $\frac{1}{3}$, independent of $d$. The probability that you get a sum outside of $[-4,4]$ is small. That means as you increase $d$, you are not causing the neurons to start out saturated so that they don't learn.

With inputs which are not normalized, those weights may not be effective at avoiding saturation.

-
So basically, one should always at least consider normalizing data.. It makes sense now. Could you explain why the std deviation will be 1/3 and how small is the probability of input sum outside the range <-4,4>? –  elmes Jan 13 '13 at 9:50
There are some basic properties of variance which imply this: If $X$ and $Y$ are independent, then $\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y)$ and if $X$ and $Y$ are independent and have mean $0$, then $\text{Var}(X*Y) = \text{Var}(X)*\text{Var}(Y)$. –  Douglas Zare Jan 14 '13 at 6:55
You can estimate the probability that a random variable is at least $12$ standard deviations away from the mean using the Chebyshev inequality. In practice this is not sharp, but the exact result depends on the distribution. –  Douglas Zare Jan 14 '13 at 6:56
By the way, I miscalculated. The variance is $\frac{1}{3}$ so the standard deviation is $\sqrt{\frac13}$. –  Douglas Zare Jan 14 '13 at 19:35
The following explanation is taken from the book: Neural Networks for Pattern Recognition by Christopher Bishop. Great book! Assume you have previously whitened the inputs to the input units, i.e. $$<x_{i}> = 0$$ and $$<x_{i}^{2}> = 1$$
You have for the activation of the units in the first layer, $$y = g(a)$$ where $$a = \sum_{i=0}^{d}w_{i}x_{i}$$. Now, since you choose the weights independently from the inputs, $$<a> = \sum_{i=0}^{d}<w_{i}x_{i}> = \sum_{i=0}^{d}<w_{i}><x_{i}> = 0$$ and $$<a^2> = \left<\left(\sum_{i=0}^{d}w_{i}x_{i}\right) \left(\sum_{i=0}^{d}w_{i}x_{i}\right)\right> = \sum_{i=0}^{d}<w_{i}^{2}><x_{i}^{2}> = \sigma^{2}d$$ where sigma is the variance of the distribution of weights. To derive this result you need to recall that weights are initialized independently from each other, i.e. $$<w_{i}w_{j}> = \delta_{ij}$$
Minor mistake: $<x_i^2> = 1$ instead of $0$. –  bayerj Jan 13 '13 at 21:30
This explains how you reach a ceratin $\sigma$ assuming you know the required $\alpha$. As I understand, $\alpha$ should be small to allow a big value of the sigmoid derivative, but not too small so that the deltas won't vanish. Is this true? If so - is it a good rule of thumb to say that $\alpha$ should be ~0.2? –  Uri Apr 23 '13 at 15:00