1
$\begingroup$

I am following deeplearning.ai's videos on Coursera. Prof Ng mentions that specific random initialisations for the weights(for example, by Xavier or He initialisations) can help optimise learning.

Particularly he says that multiplying a standard normal distribution by $\sqrt{\frac{2}{n^{[l-1]}}}$ produces a normal distribution with variance $ \frac{2}{n}$. Is there a proof for this statement? More particularly how can I derive the variance for a normal distribution multiplied by some factor?

Some background:

I attempted to find the variance for c*f(x) assuming f(x) is Gaussian and c is a constant by following the steps mentioned in the following question(particularly the accepted answer):

https://math.stackexchange.com/questions/518281/how-to-derive-the-mean-and-variance-of-a-gaussian-random-variable

However, I found some discrepancies in the results. I came to understand that this is probably caused by the fact that c*f(x) is no longer a PDF and hence we can no longer use the traditional definition of E(x), V(x) to estimate the mean and variance. So how are these particular random initialisations which lead to particular variances for the normal distribution arrived at?

Edit:

$ f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{\frac{-(x-\mu)^2}{2\sigma^2}} $

So, c*f(x) =

$\frac{\sqrt{c^2}}{\sqrt{2\pi\sigma^2}}e^{\frac{-(x-\mu)^2}{2\sigma^2}} $

I guess this is the part I don't understand. From what I can see, the constant only modifies the other term of the above expression. So, how is the new expression a valid PDF? The value of $2\sigma^2$ in the denominator of the exponent remains unchanged. So, how does this affect the variance?

Edit:

The following is an answer I got from another question.

The expressions for the mean and standard deviation are true for any random variable: $$ E(X+k) = EX+Ek = EX + k, \quad V(X+k)=VX+Vk=VX, $$ $$ E(kX) = kEX, \quad V(kX) = k^2VX $$ I am sure you can find proofs for these easily. But we still need to show that the transformed variables have normal distributions. We can do that as follows. Assume $k>0$. Then: $$ P(kX<x) = P(X<\frac xk) = \int_{-\infty}^\frac xk \frac{1}{\sqrt{2\pi}\sigma}\exp \left(-\frac{(t-\mu)^2}{2\sigma^2}\right)dt $$ $$ = \int_{-\infty}^x \frac{1}{\sqrt{2\pi}k\sigma}\exp \left(-\frac{(u -k\mu)^2}{2(k\sigma)^2}\right)du $$ where we substituted $u=kt$. This is the CDF of $\mathcal N(k\mu, k^2\sigma^2)$, so $kX$ has that distribution. (Note that we rediscovered the mean and variance). The cases $kX$ for $k<0$, and $X+k$ can be shown similarly.

$\endgroup$
  • $\begingroup$ Thank you. Sure. Will make an edit to the original question. $\endgroup$ – Nitin Oct 21 '19 at 17:13
1
$\begingroup$

There's a difference between the parameters of a distribution $F$ and a multiple of a random deviate drawn from $F$.

In your example, suppose we have $x\sim \mathcal{N}(0, 1^2)$. Then $y=cx$ for some $c>0$ has distribution $y\sim \mathcal{N}(0, c^2)$. For details, see Going from a normal distribution to a standard normal distribution with a change of variable

This is obviously not the same as $cf(x;0, \sigma^2)$; by inspection, we can see that $cf$ is not a valid distribution because it will only integrate to 1 in the special case of $c=1$.

The key detail here is that you're multiplying the standard normal deviate by $c$, not the standard normal distribution.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ That makes sense. However, I think there may be a small mistake in your formula. If $ x $ ~ $ N(\mu, \sigma^2) $, then for $y = cx$, for some c>0, the distribution is $ ~N(c\mu, c^2\sigma^2) $. $\endgroup$ – Nitin Oct 24 '19 at 17:03
  • $\begingroup$ I have added a proof I found to the original question. Could you check it and let me know. It seems to me the mean shifts. Thank you. $\endgroup$ – Nitin Oct 24 '19 at 17:25
  • $\begingroup$ Im guessing the mean doesn't shift for a standard normal distribution as it will be zero. But for other normal distributions, the mean appears to shift. $\endgroup$ – Nitin Oct 24 '19 at 17:42
  • $\begingroup$ I see now that the flaw was generalizing beyond the standard normal case. In the original question, the focus is on 0-mean normals, so the relationship holds. $\endgroup$ – Sycorax Oct 24 '19 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.