# What are good initial weights in a neural network?

I have just heard, that it's a good idea to choose initial weights of a neural network from the range $(\frac{-1}{\sqrt d} , \frac{1}{\sqrt d})$, where $d$ is the number of inputs to a given neuron. It is assumed, that the sets are normalized - mean 0, variance 1 (don't know if this matters).

Why is this a good idea?

-

I assume you are using logistic neurons, and that you are training by gradient descent/back-propagation.

The logistic function is close to flat for large positive or negative inputs. The derivative at an input of $2$ is about $1/10$, but at $10$ the derivative is about $1/22000$ . This means that if the input of a logistic neuron is $10$ then, for a given training signal, the neuron will learn about $2200$ times slower that if the input was $2$.

If you want the neuron to learn quickly, you either need to produce a huge training signal (such as with a cross-entropy loss function) or you want the derivative to be large. To make the derivative large, you set the initial weights so that you often get inputs in the range $[-4,4]$.

The initial weights you give might or might not work. It depends on how the inputs are normalized. If the inputs are normalized to have mean $0$ and standard deviation $1$, then a random sum of $d$ terms with weights uniform on $(\frac{-1}{\sqrt{d}},\frac{1}{\sqrt{d}})$ will have mean $0$ and variance $\frac{1}{3}$, independent of $d$. The probability that you get a sum outside of $[-4,4]$ is small. That means as you increase $d$, you are not causing the neurons to start out saturated so that they don't learn.

With inputs which are not normalized, those weights may not be effective at avoiding saturation.

-
So basically, one should always at least consider normalizing data.. It makes sense now. Could you explain why the std deviation will be 1/3 and how small is the probability of input sum outside the range <-4,4>? – elmes Jan 13 '13 at 9:50
There are some basic properties of variance which imply this: If $X$ and $Y$ are independent, then $\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y)$ and if $X$ and $Y$ are independent and have mean $0$, then $\text{Var}(X*Y) = \text{Var}(X)*\text{Var}(Y)$. – Douglas Zare Jan 14 '13 at 6:55
You can estimate the probability that a random variable is at least $12$ standard deviations away from the mean using the Chebyshev inequality. In practice this is not sharp, but the exact result depends on the distribution. – Douglas Zare Jan 14 '13 at 6:56
By the way, I miscalculated. The variance is $\frac{1}{3}$ so the standard deviation is $\sqrt{\frac13}$. – Douglas Zare Jan 14 '13 at 19:35

The following explanation is taken from the book: Neural Networks for Pattern Recognition by Christopher Bishop. Great book! Assume you have previously whitened the inputs to the input units, i.e. $$<x_{i}> = 0$$ and $$<x_{i}^{2}> = 1$$

The question is: how to best choose the weights?. The idea is to pick values of the weights at random following a distribution which helps the optimization process to converge to a meaningful solution.

You have for the activation of the units in the first layer, $$y = g(a)$$ where $$a = \sum_{i=0}^{d}w_{i}x_{i}$$. Now, since you choose the weights independently from the inputs, $$<a> = \sum_{i=0}^{d}<w_{i}x_{i}> = \sum_{i=0}^{d}<w_{i}><x_{i}> = 0$$ and $$<a^2> = \left<\left(\sum_{i=0}^{d}w_{i}x_{i}\right) \left(\sum_{i=0}^{d}w_{i}x_{i}\right)\right> = \sum_{i=0}^{d}<w_{i}^{2}><x_{i}^{2}> = \sigma^{2}d$$ where sigma is the variance of the distribution of weights. To derive this result you need to recall that weights are initialized independently from each other, i.e. $$<w_{i}w_{j}> = \delta_{ij}$$

-
Minor mistake: $<x_i^2> = 1$ instead of $0$. – bayerj Jan 13 '13 at 21:30
Thanks, already fixed – jpmuc Jan 14 '13 at 8:24
This explains how you reach a ceratin $\sigma$ assuming you know the required $\alpha$. As I understand, $\alpha$ should be small to allow a big value of the sigmoid derivative, but not too small so that the deltas won't vanish. Is this true? If so - is it a good rule of thumb to say that $\alpha$ should be ~0.2? – Uri Apr 23 '13 at 15:00
This is specially true for deep neural networks, where units tend to saturate quickly as you add layers. There are a number of papers dealing with that question. A good start point might be "Understanding the difficulty of training deep feedforward neural networks" by glorot and bengio – jpmuc Apr 24 '13 at 7:10

First, weights shouldn't be set to zeros in order to break the symmetry when backprogragating:

Biases can generally be initialized to zero but weights need to be initialized carefully to break the symmetry between hidden units of the same layer. Because different output units receive different gradient signals, this symmetry breaking issue does not concern the output weights (into the output units), which can therefore also be set to zero.

Some initialization strategies:

• [2] and [3] recommend scaling by the inverse of the square root of the fan-in
• Glorot and Bengio (2010) and the Deep Learning Tutorials use a combination of the fan-in and fan-out:
• for sigmoid units: sample a Uniform(-r, r) with $r=\sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).
• for hyperbolic tangent units: sample a Uniform(-r, r) with $r=4 \sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).
• in the case of RBMs, a zero-mean Gaussian with a small standard deviation around 0.1 or 0.01 works well (Hinton, 2010) to initialize the weights.
• Orthogonal random matrix initialization, i.e. W = np.random.randn(ndim, ndim); u, s, v = np.linalg.svd(W) then use u as your initialization matrix.

Also, unsupervised pre-training may help in some situations:

An important choice is whether one should use unsupervised pre-training (and which unsupervised feature learning algorithm to use) in order to initialize parameters. In most settings we have found unsupervised pre-training to help and very rarely to hurt, but of course that implies additional training time and additional hyper-parameters.

Some ANN libraries also have some interesting lists, e.g. Lasagne:

Constant([val]) Initialize weights with constant value.
Normal([std, mean]) Sample initial weights from the Gaussian distribution.
Uniform([range, std, mean]) Sample initial weights from the uniform distribution.
Glorot(initializer[, gain, c01b])   Glorot weight initialization.
GlorotNormal([gain, c01b])  Glorot with weights sampled from the Normal distribution.
GlorotUniform([gain, c01b]) Glorot with weights sampled from the Uniform distribution.
He(initializer[, gain, c01b])   He weight initialization.
HeNormal([gain, c01b])  He initializer with weights sampled from the Normal distribution.
HeUniform([gain, c01b]) He initializer with weights sampled from the Uniform distribution.
Orthogonal([gain])  Intialize weights as Orthogonal matrix.
Sparse([sparsity, std]) Initialize weights as sparse matrix.


[1] Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.

[2] LeCun, Y., Bottou, L., Orr, G. B., and Muller, K. (1998a). Efficient backprop. In Neural Networks, Tricks of the Trade.

[3] Glorot, Xavier, and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." International conference on artificial intelligence and statistics. 2010.

-
I`d like to add two useful references: 1)Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification -- about importance of activation-aware scaling arxiv.org/abs/1502.01852 2)Exact solutions to the nonlinear dynamics of learning in deep linear neural networks arxiv.org/abs/1312.6120 - orthonormal matrices are much better than just Gaussian noise – old-ufo Jan 26 at 11:31

According to Hugo Larochelle, Glorot & Bengio (2010), initialize the weights uniformly within the interval $[−b,b]$, where

$$b = \sqrt{\frac{6}{H_k + H_{k+1}}},$$

$H_k$ and $H_{k+1}$ are the sizes of the layers before and after the weight matrix.

-