36
$\begingroup$

Say I have two normal distributions A and B with means $\mu_A$ and $\mu_B$ and variances $\sigma_A$ and $\sigma_B$. I want to take a weighted mixture of these two distributions using weights $p$ and $q$ where $0\le p \le 1$ and $q = 1-p$. I know that the mean of this mixture would be $\mu_{AB} = (p\times\mu_A) + (q\times\mu_B)$.

What would the variance be?


A concrete example would be if I knew the parameters for the distribution of male and female height. If I had a room of people that was 60% male, I could produce the expected mean height for the whole room, but what about the variance?

$\endgroup$
  • $\begingroup$ Re terminology: The mixture simply has a mean and a variance; there's no sense in qualifying these as "expected," unless you are perhaps hinting that $p$ and $q$ should be considered random variables. $\endgroup$ – whuber Oct 6 '11 at 16:44
  • $\begingroup$ I know that the mixture of two gaussian distributions is identificable. But if the two distributions have the same emans? I.e:, is the mixture of two normal distributions with the same means and different standard deviations identificable? There are papers in this context? Thanks in advance $\endgroup$ – user34807 Nov 14 '13 at 11:39
  • 1
    $\begingroup$ There is a similar question with answers (dealing also with the COVARIANCES) here: math.stackexchange.com/q/195911/96547 $\endgroup$ – hplieninger Mar 17 '16 at 10:40
58
$\begingroup$

The variance is the second moment minus the square of the first moment, so it suffices to compute moments of mixtures.

In general, given distributions with PDFs $f_i$ and constant (non-random) weights $p_i$, the PDF of the mixture is

$$f(x) = \sum_i{p_i f_i(x)},$$

from which it follows immediately for any moment $k$ that

$$\mu^{(k)} = \mathbb{E}_{f}[x^k] = \sum_i{p_i \mathbb{E}_{f_i}[x^k]} = \sum_i{p_i \mu_i^{(k)}}.$$

I have written $\mu^{(k)}$ for the $k^{th}$ moment of $f$ and $\mu_i^{(k)}$ for the $k^{th}$ moment of $f_i$.

Using these formulae, the variance can be written

$$\text{Var}(f) = \mu^{(2)} - \left(\mu^{(1)}\right)^2 = \sum_i{p_i \mu_i^{(2)}} - \left(\sum_i{p_i \mu_i^{(1)}}\right)^2.$$

Equivalently, if the variances of the $f_i$ are given as $\sigma^2_i$, then $\mu^{(2)}_i = \sigma^2_i + \left(\mu^{(1)}_i\right)^2$, enabling the variance of the mixture $f$ to be written in terms of the variances and means of its components as

$$\eqalign{ \text{Var}(f) &= \sum_i{p_i \left(\sigma^2_i + \left(\mu^{(1)}_i\right)^2\right)} - \left(\sum_i{p_i \mu_i^{(1)}}\right)^2 \\ &= \sum_i{p_i \sigma^2_i} + \sum_i{p_i\left(\mu_i^{(1)}\right)^2} - \left(\sum_{i}{p_i \mu_i^{(1)}}\right)^2. }$$

In words, this is the (weighted) average variance plus the average squared mean minus the square of the average mean. Because squaring is a convex function, Jensen's Inequality asserts that the average squared mean can be no less than the square of the average mean. This allows us to understand the formula as stating the variance of the mixture is the mixture of the variances plus a non-negative term accounting for the (weighted) dispersion of the means.

In your case the variance is

$$p_A \sigma_A^2 + p_B \sigma_B^2 + \left[p_A\mu_A^2 + p_B\mu_B^2 - (p_A \mu_A + p_B \mu_B)^2\right].$$

We can interpret this is a weighted mixture of the two variances, $p_A\sigma_A^2 + p_B\sigma_B^2$, plus a (necessarily positive) correction term to account for the shifts from the individual means relative to the overall mixture mean.

The utility of this variance in interpreting data, such as given in the question, is doubtful, because the mixture distribution will not be Normal (and may depart substantially from it, to the extent of exhibiting bimodality).

$\endgroup$
  • 8
    $\begingroup$ In particular, noting that $p_A+p_B=1$, your last expression simplifies to $\sigma^2=\mu^{(2)}-\mu^2=p_A\sigma_A^2+p_B\sigma_B^2+p_Ap_B(\mu_A-\mu_B)^2$. $\endgroup$ – Ilmari Karonen Jun 20 '13 at 18:46
  • 2
    $\begingroup$ Or, if we do impose a probabilistic explanation for a mixture density (there is an event $A$ of probabiity $p_A$ and the conditional density of $X$ given $A$ is $N(\mu_A,\sigma_A^2)$ while the conditional density of $X$ given $A^c = B$ is $N(\mu_B,\sigma_B^2)$), then var$(X)$ is the sum of the mean of the conditional variance plus the variance of the conditional mean. The latter is a discrete RV $Y$ with values $\mu_A, \mu_B$ with probabilities $p$ and $q$ and your expression in square brackets is readily recognized to be $E[Y^2]-(E[Y])^2$. $\endgroup$ – Dilip Sarwate May 20 '15 at 15:25
  • 1
    $\begingroup$ @Neodyme By definition, the variance is the second moment minus the mean squared. Therefore, the second moment is the variance plus the mean squared. $\endgroup$ – whuber Feb 9 '16 at 21:05
  • 1
    $\begingroup$ @Neodyme use $E(X)=\mu$. $\endgroup$ – whuber Feb 10 '16 at 14:15
  • 1
    $\begingroup$ @Kiran Although in some cases the mixture might look Normal, it will not be. One way to see that is to compute its excess kurtosis using the formulas given here. It will be nonzero unless all the standard deviations are equal--in which case the "mixture" isn't really a mixture in the first place. $\endgroup$ – whuber Oct 11 '16 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.