67
$\begingroup$

Say I have two normal distributions A and B with means $\mu_A$ and $\mu_B$ and variances $\sigma_A$ and $\sigma_B$. I want to take a weighted mixture of these two distributions using weights $p$ and $q$ where $0\le p \le 1$ and $q = 1-p$. I know that the mean of this mixture would be $\mu_{AB} = (p\times\mu_A) + (q\times\mu_B)$.

What would the variance be?


A concrete example would be if I knew the parameters for the distribution of male and female height. If I had a room of people that was 60% male, I could produce the expected mean height for the whole room, but what about the variance?

$\endgroup$
3
  • $\begingroup$ Re terminology: The mixture simply has a mean and a variance; there's no sense in qualifying these as "expected," unless you are perhaps hinting that $p$ and $q$ should be considered random variables. $\endgroup$
    – whuber
    Commented Oct 6, 2011 at 16:44
  • $\begingroup$ I know that the mixture of two gaussian distributions is identificable. But if the two distributions have the same emans? I.e:, is the mixture of two normal distributions with the same means and different standard deviations identificable? There are papers in this context? Thanks in advance $\endgroup$
    – user34807
    Commented Nov 14, 2013 at 11:39
  • 1
    $\begingroup$ There is a similar question with answers (dealing also with the COVARIANCES) here: math.stackexchange.com/q/195911/96547 $\endgroup$ Commented Mar 17, 2016 at 10:40

3 Answers 3

92
$\begingroup$

The variance is the second moment minus the square of the first moment, so it suffices to compute moments of mixtures.

In general, given distributions with PDFs $f_i$ and constant (non-random) weights $p_i$, the PDF of the mixture is

$$f(x) = \sum_i{p_i f_i(x)},$$

from which it follows immediately for any moment $k$ that

$$\mu^{(k)} = \mathbb{E}_{f}[x^k] = \sum_i{p_i \mathbb{E}_{f_i}[x^k]} = \sum_i{p_i \mu_i^{(k)}}.$$

I have written $\mu^{(k)}$ for the $k^{th}$ moment of $f$ and $\mu_i^{(k)}$ for the $k^{th}$ moment of $f_i$.

Using these formulae, the variance can be written

$$\text{Var}(f) = \mu^{(2)} - \left(\mu^{(1)}\right)^2 = \sum_i{p_i \mu_i^{(2)}} - \left(\sum_i{p_i \mu_i^{(1)}}\right)^2.$$

Equivalently, if the variances of the $f_i$ are given as $\sigma^2_i$, then $\mu^{(2)}_i = \sigma^2_i + \left(\mu^{(1)}_i\right)^2$, enabling the variance of the mixture $f$ to be written in terms of the variances and means of its components as

$$\eqalign{ \text{Var}(f) &= \sum_i{p_i \left(\sigma^2_i + \left(\mu^{(1)}_i\right)^2\right)} - \left(\sum_i{p_i \mu_i^{(1)}}\right)^2 \\ &= \sum_i{p_i \sigma^2_i} + \sum_i{p_i\left(\mu_i^{(1)}\right)^2} - \left(\sum_{i}{p_i \mu_i^{(1)}}\right)^2. }$$

In words, this is the (weighted) average variance plus the average squared mean minus the square of the average mean. Because squaring is a convex function, Jensen's Inequality asserts that the average squared mean can be no less than the square of the average mean. This allows us to understand the formula as stating the variance of the mixture is the mixture of the variances plus a non-negative term accounting for the (weighted) dispersion of the means.

In your case the variance is

$$p_A \sigma_A^2 + p_B \sigma_B^2 + \left[p_A\mu_A^2 + p_B\mu_B^2 - (p_A \mu_A + p_B \mu_B)^2\right].$$

We can interpret this is a weighted mixture of the two variances, $p_A\sigma_A^2 + p_B\sigma_B^2$, plus a (necessarily positive) correction term to account for the shifts from the individual means relative to the overall mixture mean.

The utility of this variance in interpreting data, such as given in the question, is doubtful, because the mixture distribution will not be Normal (and may depart substantially from it, to the extent of exhibiting bimodality).

$\endgroup$
14
  • 11
    $\begingroup$ In particular, noting that $p_A+p_B=1$, your last expression simplifies to $\sigma^2=\mu^{(2)}-\mu^2=p_A\sigma_A^2+p_B\sigma_B^2+p_Ap_B(\mu_A-\mu_B)^2$. $\endgroup$ Commented Jun 20, 2013 at 18:46
  • 3
    $\begingroup$ Or, if we do impose a probabilistic explanation for a mixture density (there is an event $A$ of probabiity $p_A$ and the conditional density of $X$ given $A$ is $N(\mu_A,\sigma_A^2)$ while the conditional density of $X$ given $A^c = B$ is $N(\mu_B,\sigma_B^2)$), then var$(X)$ is the sum of the mean of the conditional variance plus the variance of the conditional mean. The latter is a discrete RV $Y$ with values $\mu_A, \mu_B$ with probabilities $p$ and $q$ and your expression in square brackets is readily recognized to be $E[Y^2]-(E[Y])^2$. $\endgroup$ Commented May 20, 2015 at 15:25
  • 1
    $\begingroup$ @Neodyme By definition, the variance is the second moment minus the mean squared. Therefore, the second moment is the variance plus the mean squared. $\endgroup$
    – whuber
    Commented Feb 9, 2016 at 21:05
  • 1
    $\begingroup$ @Neodyme use $E(X)=\mu$. $\endgroup$
    – whuber
    Commented Feb 10, 2016 at 14:15
  • 1
    $\begingroup$ @Kiran Although in some cases the mixture might look Normal, it will not be. One way to see that is to compute its excess kurtosis using the formulas given here. It will be nonzero unless all the standard deviations are equal--in which case the "mixture" isn't really a mixture in the first place. $\endgroup$
    – whuber
    Commented Oct 11, 2016 at 18:38
8
$\begingroup$

The solution of whuber is perfect but it seems that something lacks to join this result with the LTV (law of total variance). The previous result $$\sigma^2=p_A \sigma_A^2+p_B \sigma_B^2+p_A \mu_A^2+p_B \mu_B^2−\mu^2$$

can be rewritten taking into account that $2p_A\mu_A\mu +2p_B\mu_B\mu=2\mu(p_A\mu_A+p_B\mu_B)=2\mu^2$, so

$$\sigma^2=p_A \sigma_A^2+p_B \sigma_B^2+p_A \mu_A^2+p_B \mu_B^2+\mu^2 -2p_A\mu_A\mu -2p_B\mu_B\mu$$

and finally

$$\sigma^2=p_A \sigma_A^2+p_B \sigma_B^2+p_A (\mu_A - \mu)^2+p_B(\mu_B-\mu)^2$$ what is the LTV typical expression that we are used to see.

$\endgroup$
3
$\begingroup$

The solution of whuber is perfect. I just want to add that the term in the square brackets has another nice and simple expression, so

$$\sigma^2=p_A\sigma_A^2+p_B\sigma_B^2+p_Ap_B(\mu_A-\mu_B)^2.$$

$\endgroup$
3
  • $\begingroup$ This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review $\endgroup$
    – utobi
    Commented Feb 9, 2023 at 16:29
  • 2
    $\begingroup$ Thank you for this note @utobi. I have been reading stats.stackexchange for years but am new to posting. I am sorry I don't really understand why this is not an appropriate post. The question asked what is the variance. whuber derived the variance. JGiner gave another formula that is helpful. I just did the exact same thing, providing another formula which I think is helpful. I hope you don't mind explaining so I understand. I would like to be a good member of the community. Thank you! $\endgroup$ Commented Feb 9, 2023 at 16:49
  • $\begingroup$ This simple expression was already given by Ilmari Karonen Jun 20, 2013 as a comment. It's not an answer. $\endgroup$
    – stweb
    Commented May 8 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.