What is the variance of the weighted mixture of two gaussians?

Question

Say I have two normal distributions A and B with means $\mu_A$ and $\mu_B$ and variances $\sigma_A$ and $\sigma_B$. I want to take a weighted mixture of these two distributions using weights $p$ and $q$ where $0\le p \le 1$ and $q = 1-p$. I know that the mean of this mixture would be $\mu_{AB} = (p\times\mu_A) + (q\times\mu_B)$.

What would the variance be?

---

A concrete example would be if I knew the parameters for the distribution of male and female height. If I had a room of people that was 60% male, I could produce the expected mean height for the whole room, but what about the variance?

Answer 1

The variance is the second moment minus the square of the first moment, so it suffices to compute moments of mixtures.

In general, given distributions with PDFs $f_i$ and constant (non-random) weights $p_i$, the PDF of the mixture is

$$f(x) = \sum_i{p_i f_i(x)},$$

from which it follows immediately for any moment $k$ that

$$\mu^{(k)} = \mathbb{E}_{f}[x^k] = \sum_i{p_i \mathbb{E}_{f_i}[x^k]} = \sum_i{p_i \mu_i^{(k)}}.$$

I have written $\mu^{(k)}$ for the $k^{th}$ moment of $f$ and $\mu_i^{(k)}$ for the $k^{th}$ moment of $f_i$.

Using these formulae, the variance can be written

$$\text{Var}(f) = \mu^{(2)} - \left(\mu^{(1)}\right)^2 = \sum_i{p_i \mu_i^{(2)}} - \left(\sum_i{p_i \mu_i^{(1)}}\right)^2.$$

Equivalently, if the variances of the $f_i$ are given as $\sigma^2_i$, then $\mu^{(2)}_i = \sigma^2_i + \left(\mu^{(1)}_i\right)^2$, enabling the variance of the mixture $f$ to be written in terms of the variances and means of its components as

$$\eqalign{ \text{Var}(f) &= \sum_i{p_i \left(\sigma^2_i + \left(\mu^{(1)}_i\right)^2\right)} - \left(\sum_i{p_i \mu_i^{(1)}}\right)^2 \\ &= \sum_i{p_i \sigma^2_i} + \sum_i{p_i\left(\mu_i^{(1)}\right)^2} - \left(\sum_{i}{p_i \mu_i^{(1)}}\right)^2. }$$

In words, this is the (weighted) average variance plus the average squared mean minus the square of the average mean. Because squaring is a convex function, Jensen's Inequality asserts that the average squared mean can be no less than the square of the average mean. This allows us to understand the formula as stating the variance of the mixture is the mixture of the variances plus a non-negative term accounting for the (weighted) dispersion of the means.

In your case the variance is

$$p_A \sigma_A^2 + p_B \sigma_B^2 + \left[p_A\mu_A^2 + p_B\mu_B^2 - (p_A \mu_A + p_B \mu_B)^2\right].$$

We can interpret this is a weighted mixture of the two variances, $p_A\sigma_A^2 + p_B\sigma_B^2$, plus a (necessarily positive) correction term to account for the shifts from the individual means relative to the overall mixture mean.

The utility of this variance in interpreting data, such as given in the question, is doubtful, because the mixture distribution will not be Normal (and may depart substantially from it, to the extent of exhibiting bimodality).

Answer 2

The solution of whuber is perfect but it seems that something lacks to join this result with the LTV (law of total variance). The previous result $$\sigma^2=p_A \sigma_A^2+p_B \sigma_B^2+p_A \mu_A^2+p_B \mu_B^2−\mu^2$$

can be rewritten taking into account that $2p_A\mu_A\mu +2p_B\mu_B\mu=2\mu(p_A\mu_A+p_B\mu_B)=2\mu^2$, so

$$\sigma^2=p_A \sigma_A^2+p_B \sigma_B^2+p_A \mu_A^2+p_B \mu_B^2+\mu^2 -2p_A\mu_A\mu -2p_B\mu_B\mu$$

and finally

$$\sigma^2=p_A \sigma_A^2+p_B \sigma_B^2+p_A (\mu_A - \mu)^2+p_B(\mu_B-\mu)^2$$ what is the LTV typical expression that we are used to see.

Answer 3

The solution of whuber is perfect. I just want to add that the term in the square brackets has another nice and simple expression, so

$$\sigma^2=p_A\sigma_A^2+p_B\sigma_B^2+p_Ap_B(\mu_A-\mu_B)^2.$$