# How to get Variance from Gaussian distribution and Random Initialization

I am following deeplearning.ai's videos on Coursera. Prof Ng mentions that specific random initialisations for the weights(for example, by Xavier or He initialisations) can help optimise learning.

Particularly he says that multiplying a standard normal distribution by $$\sqrt{\frac{2}{n^{[l-1]}}}$$ produces a normal distribution with variance $$\frac{2}{n}$$. Is there a proof for this statement? More particularly how can I derive the variance for a normal distribution multiplied by some factor?

Some background:

I attempted to find the variance for c*f(x) assuming f(x) is Gaussian and c is a constant by following the steps mentioned in the following question(particularly the accepted answer):

https://math.stackexchange.com/questions/518281/how-to-derive-the-mean-and-variance-of-a-gaussian-random-variable

However, I found some discrepancies in the results. I came to understand that this is probably caused by the fact that c*f(x) is no longer a PDF and hence we can no longer use the traditional definition of E(x), V(x) to estimate the mean and variance. So how are these particular random initialisations which lead to particular variances for the normal distribution arrived at?

Edit:

$$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{\frac{-(x-\mu)^2}{2\sigma^2}}$$

So, c*f(x) =

$$\frac{\sqrt{c^2}}{\sqrt{2\pi\sigma^2}}e^{\frac{-(x-\mu)^2}{2\sigma^2}}$$

I guess this is the part I don't understand. From what I can see, the constant only modifies the other term of the above expression. So, how is the new expression a valid PDF? The value of $$2\sigma^2$$ in the denominator of the exponent remains unchanged. So, how does this affect the variance?

Edit:

The following is an answer I got from another question.

The expressions for the mean and standard deviation are true for any random variable: $$E(X+k) = EX+Ek = EX + k, \quad V(X+k)=VX+Vk=VX,$$ $$E(kX) = kEX, \quad V(kX) = k^2VX$$ I am sure you can find proofs for these easily. But we still need to show that the transformed variables have normal distributions. We can do that as follows. Assume $$k>0$$. Then: $$P(kX $$= \int_{-\infty}^x \frac{1}{\sqrt{2\pi}k\sigma}\exp \left(-\frac{(u -k\mu)^2}{2(k\sigma)^2}\right)du$$ where we substituted $$u=kt$$. This is the CDF of $$\mathcal N(k\mu, k^2\sigma^2)$$, so $$kX$$ has that distribution. (Note that we rediscovered the mean and variance). The cases $$kX$$ for $$k<0$$, and $$X+k$$ can be shown similarly.

• Thank you. Sure. Will make an edit to the original question. Oct 21, 2019 at 17:13

There's a difference between the parameters of a distribution $$F$$ and a multiple of a random deviate drawn from $$F$$.

In your example, suppose we have $$x\sim \mathcal{N}(0, 1^2)$$. Then $$y=cx$$ for some $$c>0$$ has distribution $$y\sim \mathcal{N}(0, c^2)$$. For details, see Going from a normal distribution to a standard normal distribution with a change of variable

This is obviously not the same as $$cf(x;0, \sigma^2)$$; by inspection, we can see that $$cf$$ is not a valid distribution because it will only integrate to 1 in the special case of $$c=1$$.

The key detail here is that you're multiplying the standard normal deviate by $$c$$, not the standard normal distribution.

• That makes sense. However, I think there may be a small mistake in your formula. If $x$ ~ $N(\mu, \sigma^2)$, then for $y = cx$, for some c>0, the distribution is $~N(c\mu, c^2\sigma^2)$. Oct 24, 2019 at 17:03
• I have added a proof I found to the original question. Could you check it and let me know. It seems to me the mean shifts. Thank you. Oct 24, 2019 at 17:25
• Im guessing the mean doesn't shift for a standard normal distribution as it will be zero. But for other normal distributions, the mean appears to shift. Oct 24, 2019 at 17:42
• I see now that the flaw was generalizing beyond the standard normal case. In the original question, the focus is on 0-mean normals, so the relationship holds.
– Sycorax
Oct 24, 2019 at 18:42