Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
weight initialization is important for modern deep learning. To understand [1,2], I would like to understand the following:
$$ E[x^2] = 0.5 Var[y], $$
where $x= max(0,y)$, $E[.]$ is the expectation, $Var[.]$ the variance, $x,y$ are random variables. We assume $y$ to have zero mean and to be symmetrical around the mean.
Thanx for an explanation/derivation
K
[1] http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf
In terms of integrals you have:
$$ E[x^2] = \int_{-\infty}^{+\infty} \max(0,y)^2 p(y) dy $$
where the part $y < 0$ does not contribute to the Integral
$$ = \int_{0}^{+\infty} y^2 p(y) dy $$
which we can write as half the integral over the entire real domain ($y^2$ is symmetric around 0 and $p(y)$ is assumed to be symmetric around $0$):
$$ = \frac{1}{2}\int_{-\infty}^{+\infty} y^2 p(y) dy $$
now subtracting zero in the square we get:
$$ = \frac{1}{2}\int_{-\infty}^{+\infty} (y - E[y])^2 p(y) dy $$
which is
$$ = \frac{1}{2} E[(y - E[y])^2] = \frac{1}{2} Var[y] $$
If $y$ is symmetric around zero the following holds: $$ V[y] = E[y^2] - E[y^2] = E[y^2] = E[I(y<0)y^2]+E[I(y>0)y^2] = 2E[I(y>0)y^2] $$ Thus, $$ E[x^2] = E[\max(0, y)^2] = E[I(y>0)y^2] = \frac{1}{2}E[y^2] = \frac{1}{2}V[y]. $$
The statement is not true in general if $y$ is not symmetric around zero.
