Do frameworks (e.g. Keras) use the wrong distribution for Glorot/Xavier weight initialization?

Question

When looking at the paper describing the Glorot/Xavier uniform weight initialization, the weights are sampled according to a uniform distribution according to equation 16

$$W \sim U[-\frac{\sqrt{6}}{\sqrt{n_j+n_{j+1}}}, \frac{\sqrt{6}}{\sqrt{n_j+n_{j+1}}}]$$

If I interpret the paper correctly, $n_j$ is the number of neurons in the current layer, and $n_{j+1}$ is the number of neurons in the next layer.

Looking at the implementation in Keras the bounds of the uniform distribution are calculated using a square root over fan_in + fan_out and fan_in is the numbers of neurons in the previous layer, and fan_out is the number of neurons in the current layer. So the implementation seems to be going in the "opposite" direction.

Can someone explain why you are allowed to use the opposite direction? I assume this is done because it is just difficult to get the number of neurons of the next layer when doing an initialization locally in the layer itself (since it is not known what the next layer will be).

Answer 1

The implementation is correct. There is no concept of the neurons in the typical library. Layers with parameters store weights that connect to layers of neurons, but the neuron is just an abstraction. In the simple case, lets say you have 4 input neurons and 3 output neurons. For this case, you will have 4x3 weight matrix. The shape of the matrix tells you number of input neurons and number of output. Layer that initializes values for the weights already knows all the values that it needs to know.

Layer that initializes weights has already pre-specified output shape, it just needs to know output shape from the previous layer in order to know the shape of the weight matrix.