I am currently reading Make your Own Neural Network by Tariq Rashid.
He explains that instead of choosing weights randomly at a range of -1.0 to 1.0, initial weights should be in the range $ {1 \over \sqrt{n}}$ where n= # of hidden nodes in next hidden node layer. Why?
The whole point of that initialisation scheme (sometimes called "Xavier initialisation") is to mitigate the problem of disappearing gradients caused by the form of many activation functions.
Consider the common sigmoidal activation function $s(z) = \frac{1}{1+e^{-z}} = \frac{e^z}{e^z + 1}$. Its gradient $\sigma'(z) = s(z)[1-s(z)]$ approaches $0$ away from its mean of $0$:
If this happens, it is one of many situations where a vanishing gradient can occur. And it will occur if the input to the activation function $z = \boldsymbol{w}^T \boldsymbol{x}$ lies far enough away from its mean that the gradient of its activation $s'(z)$ is $0$.
Let's say your inputs $x_i$ have already been scaled to be centred at $0$ mean with variance $1$. This means these input values $x_i$ will typically lie within the range $[-2, +2]$. (To see this, consider that for a standard Normal approximation, samples will lie within $\pm 2$ standard deviations from the mean $95\%$ of the time.)
Making a few simplifying i.i.d. assumptions and recalling that the inputs have unit variance, the variance of the activation inputs $z_i$ works out as $\text{Var}(z_i) = n \text{Var}(w_i)$.
So by initialising the weights $w_i$ with samples of variance $\frac{1}{\sqrt{n}}$, activation inputs have $\text{Var}(z_i)=1$, which keeps them away from regions where the gradient might vanish.
I think the basic idea is that if you have a bunch of nodes $n_i$, each neuron will have a lot of incoming connections. Therefore the number passed to each neuron will be large because the sum will have many elements.
$$z =\sum{w_in_i} $$
If you make the weights $w$ inversely proportional to the number of incoming neurons, the number $z$ passed to each neuron will be within a reasonable range.
Why does it matter?
If you an enormous input $z$ through a logistic activation function, the output $a(z)$ is near 0 or 1. The derivative is near 0 because the logistic function is almost flat as $|z|$ is large.
(http://deeplearning.net/software/theano/tutorial/examples.html)
If the gradient is very small, the weights will be adjusted by a very small amount in gradient descent, and training will take a long time.
For a shallow network, of only a few layers, standard deviation of the Normal distribution with a mean of 0.0, can be 0.01, or even 1.0. But, that kind of a "fixed" standard deviation is not okay for deep neural networks (NN), that have many layers. If you want to use the Uniform distribution, then it translates to the range of [-0.01, 0.01], or [-1.0, 1.0]. I'll use Normal distribution in this answer.
Let's take $\tanh$ as activation function. What you'd get in a deep NN with that initialization, in the case of $\sigma = 0.01$, is the so-called "vanishing gradients" problem. Deep layers would have all activations close to zero, and that's not good for backward pass. Namely, deep layers would have $\sigma$ near zero, because you keep multiplying by small values layer after layer, and thus all activations come close to zero. In the backward pass, we now have gradients that are close to zero in all layers, so they don't flow, and the network is unable to learn.
We can try to fix that by using a larger value for $\sigma$, say 1.0. But, now all neurons become saturated. Their activations will be either -1.0, or +1.0. What's the gradient of a constant? Zero.
The answer is somewhere in between, but it's not so easy to find. If a neuron is connected to more neurons from the previous layer, its output variance will be larger than when it's connected to fewer neurons. If we divide its weight vector by $\sqrt{n}$, we normalize its $\sigma$ to 1. I would just say that n is the number of inputs to the layer we're initializing, i.e., its number of rows, or number of columns of the previous layer, and not number of outputs of the current layer, as you said. This makes all neurons in a network have approximately the same output distribution in the beginning. This is a heuristic, that's been found to work well in practice. It helps the convergence process. So, we don't fix our range to a particular value, but rather scale initial $\sigma$ of each neuron to 1.
But, this doesn't work for the ReLU activation function. Using ReLU units leads to $\sigma$ being close to 0 in deep layers, thus making activations being close to 0, which in turn leads to gradients being close to 0. This means we have to compensate for that, by dividing weights by $\sqrt{n/2}$ instead of by $\sqrt{n}$. Then, it works fine.
Xavier Glorot et al. proposed a similar heuristic, in 2010.: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf. For example, TensorFlow has it: tf.contrib.layers.xavier_initializer.
References:
The references explain this in more detail, so if you're interested, you can check them out. I tried to be concise.
A blog post providing a good intuitive explanation of the 1/$\sqrt{N}$ result along with a simple proof for the linear case case can be found in
http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization
A bit of shameless self promotion :D...A general version of these results, along with several related ones can be found in
