1
$\begingroup$

I have always assumed that this is fact : but are there any mathematical theorems that state: finite sums of gaussian distributions are gaussian themselves?

enter image description here

In the above picture, does p(x) have a gaussian distribution? Do the sums need to be linear? Can they be non-linear?

I saw the following question: Is the joint distribution of two linear combinations of Gaussians still a multivariate normal? - but no reference/proof was provided.

Thanks

$\endgroup$
2
  • 2
    $\begingroup$ Mixtures of normal distributions need not be normal. $\endgroup$
    – BruceET
    Oct 15 '21 at 3:42
  • 1
    $\begingroup$ the whole point of mixtures is to model non-Gaussian distributions $\endgroup$
    – Aksakal
    Oct 15 '21 at 20:17
3
$\begingroup$

It is the linear combination of jointly Gaussian random variables (RVs) that results in another RV with Gaussian density. In your question, you have linear combination of Gaussian densities; therefore, the resulting density need not be Gaussian.

Below is given a working proof of this theorem. The characteristic function of an RV $X$ is $$\phi_X(t)=E(e^{\iota tx}),$$ where $E$ denotes expectation. The characteristic function of the linear combination $Y$ of two RVs $X_1$ and $X_2$ is $$Y=\alpha_1X_1+\alpha_2X_2,$$ and the characteristic function of $Y$ is $$\phi_Y(t)=E(e^{\iota tY})=E(e^{\iota t\alpha_1X_1 +\iota t\alpha_2X_2}).$$ If $X_1$ and $X_2$ are independent $\phi_Y(t)$ can be written as $$\phi_Y(t) = E(e^{\iota t\alpha_1X_1})E(e^{\iota t\alpha_2X_2}).$$

Now, for a Gaussian RV $X$ with mean $\mu$ and variance $\sigma^2$,$$\phi_X(t) = \exp(\iota t\mu-\frac{1}{2}\sigma^2t^2).$$ It is a bit tideous to derive this result but it can be done with some effort. Following this result, $$\phi_Y(t)=\exp[\iota t(\alpha_1\mu_1+\alpha_2\mu_2)-\frac{1}{2}(\alpha_1^2\sigma_1^2+\alpha_2^2\sigma_2^2)t^2].$$

Now, we also know that characteristic function of a probability distribution is unique (but it is difficult to prove). Following this result, characteristic function of $Y$ is clearly that of a Gaussian with mean $\alpha_1\mu_1 + \alpha_2\mu_2$ and variance $\alpha_1^2\sigma_1^2 + \alpha_2^2\sigma_2^2$. It basically proves that linear combination of two independent Gaussian RVs is a Gaussian RV.

Edit: The proof above is valid only when $X_1$ and $X_2$ are independent. But a similar proof can be carried out when $X_1$ and $X_2$ are not independent. Suppose $X = [X_1,X_2]^T$ is a jointly Gaussian with mean vector $\mu = [\mu_1,\mu_2]^T $ and covariance matrix $C$. Also, the characteristic function of a random vector $X$ is $$\phi_X(t_1,t_2)=E(e^{\iota t^TX})=E(e^{\iota (t_1X_1+t_2X_2)}),$$

where $t=[t_1,t_2]^T$. The characteristic function of an RV $Y=\alpha_1X_1+\alpha_2X_2$ is $$\phi_Y(u)=E(e^{\iota uY})=E(e^{\iota u(\alpha_1X_1+\alpha_2X_2)})$$ $$=E(e^{\iota (u\alpha_1X_1+u\alpha_2X_2)}).$$

Notice that, $$\phi_Y(u)=\phi_X(\alpha_1u,\alpha_2u).$$

Now, we know that (the proof is a bit involved): $$\phi_X(t)=\exp[\iota t^T\mu-\frac{1}{2}t^TCt],$$ which implies $$\phi_Y(u)=\exp[\iota u\mu_y-\frac{1}{2}u^2\sigma_y^2],$$ where $$\mu_y=\alpha^T\mu=\alpha_1\mu_1+\alpha_2\mu_2,$$ $$\sigma_y^2=\alpha^TC\alpha$$ $$\alpha=[\alpha_1,\alpha_2]^T.$$

The last four equations show that $Y$ is a Gaussian RV with mean $\mu_y$ and variance $\sigma_y^2$. I have not given the derivation of the last four equations, let me know if that would be helpful.

A side note: The RVs $X_1$ and $X_2$ should be jointly Gaussian, otherwise this result would not hold. An illustrative example is given by Chris Huang in comments section.

$\endgroup$
5
  • 1
    $\begingroup$ The first statement is not always true, it depends on their joint distribution. The proof that follows is for a very special case (independence). $\endgroup$
    – Chris Haug
    Oct 15 '21 at 12:08
  • $\begingroup$ @ChrisHaug I will add the proof for a more general case. But what are the conditions on join distribution? Can you please elaborate? $\endgroup$ Oct 15 '21 at 16:03
  • 1
    $\begingroup$ It's true if they are jointly Gaussian. You can have distributions which have Gaussian marginals but are not jointly Gaussian. For example, my answer here: stats.stackexchange.com/questions/238226 . There, $X+Y$ has an atom at zero and so cannot be Gaussian, for example. $\endgroup$
    – Chris Haug
    Oct 15 '21 at 17:21
  • 1
    $\begingroup$ @ AbhinavGupta : Thank you so much for your answer! So just to clarify: linear combinations of gaussian densities WILL NOT NECESSAIRLY have a gaussian distribution ... BUT linear combinations of gaussian random variables WILL NECESSAIRLY have a gaussian distribution? Thank you so much! $\endgroup$
    – stats555
    Oct 15 '21 at 17:31
  • $\begingroup$ @stats555 (1) No, the linear combinations of Gaussian densities are not necessarily Gaussian. (2) Linear combinations of JOINTLY Gaussian RVs is necessarily Gaussian. The conditions 'jointly' is important (As Chris Huang has pointed out). I will edit my answer to include this condition. $\endgroup$ Oct 15 '21 at 20:31
1
$\begingroup$

Suppose that you have two random variables $x\sim F_x$ and $y\sim F_y$ where $F_x,F_y$ are probability distributions. Then generally $w_xx+w_yy$ is not from $w_xF_x+w_yF_y$, even if when $F_x,F_y$ are normal distributions.

The characteristic function of the weighted sum is the product of characteristic functions of these distributions: $\psi_{w_xx+w_yy}(t)=\psi_x(w_xt)\times \psi_y(w_yt)$. You can obtain the distribution $F_{w_xx+w_yy}$ from its characteristic function, of course.

Finally, the whole idea of the Gaussian mixture is to model unknown distributions, such as those with fat tails in finance.

Now, what is meant by $w_x\mathcal N+w_y\mathcal N$? In case of a mixture it is the weighted average density (PDF).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.