Neural network weight initialization? I'm trying to understand neural network weight initialization but i need help to decipher the language people use to describe fan-in.
"N (where n is the number of the neurons inputs)"... or this definition... fan-in (the number of connections feeding into the node)", however in the Xavier Glorot paper http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2010_GlorotB10.pdf "n is the size of the previous layer (the number
of columns of W)" which sounds to me like biases are excluded.
So, does "N" aka? "fan-in" include bias connections or not.
For example, consider a neuron whose inputs are two other neurons and a bias node from the previous layer. Is the fan-in two or three?
 A: TL;DR: Biases are not included. So, N / fan_in will be 2 in your case.
Explanation: Note that Glorot and Bengio 2010 base their initializations on the variances of the weights. They assume $\mathbb V[z^i] \approx \mathbb V[x] \prod_{i'=0}^{i-1} n_{i'}\mathbb V[W^{i'}]$. This derivation comes from the approximating assumption that $z^i = f(s^i) \approx s^i = z^{i-1}W^i+b^i $ [which is true for certain symmetric activation functions around the 0, like the $\tanh$]. Since they assume that the variances of each element in the weight matrix are the same, and likewise for the inputs, we get:
$$\mathbb V[z^1]=\mathbb V[f(s^1)]=\mathbb V[f(xW^1+b^1)]\approx \mathbb V[xW^1+b^1] = \mathbb V[xW^1]
$$
The last equality holds because adding constant to a random variable doesn't change the variance (we're setting $b$ to 0, so it's a constant). Since all elements in the vector and matrix have the same variances [denoted $\mathbb V[x], \mathbb V[W]$ respectively], for each activation unit of the 1st layer, the variance will be equal to $=\mathbb V[x] n_0 \mathbb V[W]$, where $n_0$ is just the dimensionality of $x$ without the bias.
