Glorot/ Xavier Init: for sigmoid and tanh? My question is about Xavier Glorot Init. The assumptions that they make are that they approximate the activation function linearly, that this function has f'(0) = 1 and that we set the bias to 0, as well as that the input features are normalized (because all inputs should have the same variance).
As far as I understand, these assumptions hold for sigmoid and tanh activations. So does this initialization work for both equally well or is there something to consider when using the one or the other?
 A: It isn't meant for the logistic sigmoid.
The function should be symmetric with respect to the $y$-axis for equation (5) to hold. The variance calculations are so simple there because they assume that all intermediate random variables have zero mean. If we use the logistic sigmoid, we would have to apply it by adding 0.5 to each intermediate variable, which would break the zero mean assumption.
A: The variance derivation they have assumes that approximately $z^i = f(s^i) \approx s^i = z^{i-1}W^{i-1}+b^{i-1}$. This only happens for the symmetric activations, that $f(s^i) \approx s^i$. It doesn't happen for the sigmoid. Here's an image to illustrate this:

You can see that $f(x)=x$ is very close to $f(x)=\tanh(x)$ in the region around $x=0$. $f(x)=\sigma(x)$ is nowhere close.
In addition, as freegoods mentioned:
$$\mathbb V[z^i] \approx \mathbb V[z^{i-1}W^{i-1}+b^{i-1}] = \mathbb V[z^{i-1}W^{i-1}] = \\
\mathbb V[z^{i-1}]\mathbb V[W^{i-1}] + \mathbb E[z^{i-1}]^2\mathbb V[W^{i-1}] + \mathbb E[W^{i-1}]^2\mathbb V[z^{i-1}] = \\
\mathbb V[z^{i-1}]\mathbb V[W^{i-1}] + \mathbb E[z^{i-1}]^2\mathbb V[W^{i-1}] + 0\cdot\mathbb V[z^{i-1}]$$
If also $\mathbb E[z^{i-1}]$ (which again, only happens for $\tanh$ and other symmetric activation functions) then this is equal to $\mathbb V[z^{i-1}]\mathbb V[W^{i-1}]$ which is what they used in their derivation.
