# Variance calculation RELU function (deep learning)

weight initialization is important for modern deep learning. To understand [1,2], I would like to understand the following:

$$E[x^2] = 0.5 Var[y],$$

where $x= max(0,y)$, $E[.]$ is the expectation, $Var[.]$ the variance, $x,y$ are random variables. We assume $y$ to have zero mean and to be symmetrical around the mean.

Thanx for an explanation/derivation

K

In terms of integrals you have:

$$E[x^2] = \int_{-\infty}^{+\infty} \max(0,y)^2 p(y) dy$$

where the part $y < 0$ does not contribute to the Integral

$$= \int_{0}^{+\infty} y^2 p(y) dy$$

which we can write as half the integral over the entire real domain ($y^2$ is symmetric around 0 and $p(y)$ is assumed to be symmetric around $0$):

$$= \frac{1}{2}\int_{-\infty}^{+\infty} y^2 p(y) dy$$

now subtracting zero in the square we get:

$$= \frac{1}{2}\int_{-\infty}^{+\infty} (y - E[y])^2 p(y) dy$$

which is

$$= \frac{1}{2} E[(y - E[y])^2] = \frac{1}{2} Var[y]$$

If $$y$$ is symmetric around zero the following holds: $$V[y] = E[y^2] - E[y^2] = E[y^2] = E[I(y<0)y^2]+E[I(y>0)y^2] = 2E[I(y>0)y^2]$$ Thus, $$E[x^2] = E[\max(0, y)^2] = E[I(y>0)y^2] = \frac{1}{2}E[y^2] = \frac{1}{2}V[y].$$

The statement is not true in general if $$y$$ is not symmetric around zero.