weight initialization is important for modern deep learning. To understand [1,2], I would like to understand the following:

$$ E[x^2] = 0.5 Var[y], $$

where $x= max(0,y)$, $E[.]$ is the expectation, $Var[.]$ the variance, $x,y$ are random variables. We assume $y$ to have zero mean and to be symmetrical around the mean.

Thanx for an explanation/derivation


[1] http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf

[2] http://arxiv.org/abs/1502.01852


2 Answers 2


In terms of integrals you have:

$$ E[x^2] = \int_{-\infty}^{+\infty} \max(0,y)^2 p(y) dy $$

where the part $y < 0$ does not contribute to the Integral

$$ = \int_{0}^{+\infty} y^2 p(y) dy $$

which we can write as half the integral over the entire real domain ($y^2$ is symmetric around 0 and $p(y)$ is assumed to be symmetric around $0$):

$$ = \frac{1}{2}\int_{-\infty}^{+\infty} y^2 p(y) dy $$

now subtracting zero in the square we get:

$$ = \frac{1}{2}\int_{-\infty}^{+\infty} (y - E[y])^2 p(y) dy $$

which is

$$ = \frac{1}{2} E[(y - E[y])^2] = \frac{1}{2} Var[y] $$


If $y$ is symmetric around zero the following holds: $$ V[y] = E[y^2] - E[y^2] = E[y^2] = E[I(y<0)y^2]+E[I(y>0)y^2] = 2E[I(y>0)y^2] $$ Thus, $$ E[x^2] = E[\max(0, y)^2] = E[I(y>0)y^2] = \frac{1}{2}E[y^2] = \frac{1}{2}V[y]. $$

The statement is not true in general if $y$ is not symmetric around zero.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.