I am following deeplearning.ai's videos on Coursera. Prof Ng mentions that specific random initialisations for the weights(for example, by Xavier or He initialisations) can help optimise learning.
Particularly he says that multiplying a standard normal distribution by $\sqrt{\frac{2}{n^{[l-1]}}}$ produces a normal distribution with variance $ \frac{2}{n}$. Is there a proof for this statement? More particularly how can I derive the variance for a normal distribution multiplied by some factor?
Some background:
I attempted to find the variance for c*f(x) assuming f(x) is Gaussian and c is a constant by following the steps mentioned in the following question(particularly the accepted answer):
However, I found some discrepancies in the results. I came to understand that this is probably caused by the fact that c*f(x) is no longer a PDF and hence we can no longer use the traditional definition of E(x), V(x) to estimate the mean and variance. So how are these particular random initialisations which lead to particular variances for the normal distribution arrived at?
Edit:
$ f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{\frac{-(x-\mu)^2}{2\sigma^2}} $
So, c*f(x) =
$\frac{\sqrt{c^2}}{\sqrt{2\pi\sigma^2}}e^{\frac{-(x-\mu)^2}{2\sigma^2}} $
I guess this is the part I don't understand. From what I can see, the constant only modifies the other term of the above expression. So, how is the new expression a valid PDF? The value of $2\sigma^2$ in the denominator of the exponent remains unchanged. So, how does this affect the variance?
Edit:
The following is an answer I got from another question.
The expressions for the mean and standard deviation are true for any random variable: $$ E(X+k) = EX+Ek = EX + k, \quad V(X+k)=VX+Vk=VX, $$ $$ E(kX) = kEX, \quad V(kX) = k^2VX $$ I am sure you can find proofs for these easily. But we still need to show that the transformed variables have normal distributions. We can do that as follows. Assume $k>0$. Then: $$ P(kX<x) = P(X<\frac xk) = \int_{-\infty}^\frac xk \frac{1}{\sqrt{2\pi}\sigma}\exp \left(-\frac{(t-\mu)^2}{2\sigma^2}\right)dt $$ $$ = \int_{-\infty}^x \frac{1}{\sqrt{2\pi}k\sigma}\exp \left(-\frac{(u -k\mu)^2}{2(k\sigma)^2}\right)du $$ where we substituted $u=kt$. This is the CDF of $\mathcal N(k\mu, k^2\sigma^2)$, so $kX$ has that distribution. (Note that we rediscovered the mean and variance). The cases $kX$ for $k<0$, and $X+k$ can be shown similarly.