This is a homework problem. If I could have some hints on how to solve it that would be great.

Mean and Covariance of a Mixture of Two Distributions:

$$ I = \left\{ \begin{array}{l l} 1 & \quad \text{if subject in group 1}\\ 0 & \quad \text{if subject in group 2} \end{array} \right.$$

Suppose $\mathbf{Z}=(Z_1,\ldots,Z_p)^T$ is a vector of measurements on a subject with mean and covariances: $$ E[\mathbf{Z}|I=1]=\mathbf{\mu}_1, \quad E[\mathbf{Z}|I=0]=\mathbf{\mu}_2\\Cov[\mathbf{Z}|I=1]=\mathbf{\Sigma}_1, \quad Cov[\mathbf{Z}|I=0]=\mathbf{\Sigma}_2$$

Let $p=P(I=1)$. Find $E[\mathbf{Z}]$ and $Cov[\mathbf{Z}]$.

My Attempt:

(This makes sense to me) $E[\mathbf{Z}] = E[E[\mathbf{Z}|I]] = E[p\mathbf{\mu}_1+(1-p)\mathbf{\mu}_2] = p\mathbf{\mu}_1+(1-p)\mathbf{\mu}_2$.

(Not sure about this one) $$ \begin{align*}Cov[\mathbf{Z}] &= Cov[E[\mathbf{Z}|I]] + E[Cov[\mathbf{Z}|I]]\\&= Cov[p\mathbf{\mu}_1+(1-p)\mathbf{\mu}_2] + E[Cov[\mathbf{Z}|I]]\\&= ???\end{align*}$$

  • 2
    $\begingroup$ The start of a solution is outlined at stats.stackexchange.com/questions/32699, where the question (implicitly) suggests computing the raw moments of the mixture and combining them (according to the definitions) to obtain the variances and covariances. $\endgroup$ – whuber Sep 6 '13 at 18:33
  • $\begingroup$ Thanks, I've gotten it figured out using $Cov[Z]=E[ZZ^t]-EZ(EZ)^T$ and some hints from this math.stackexchange.com/questions/147497/… $\endgroup$ – bdeonovic Sep 6 '13 at 19:12

Here is my solution if anyone is curious (sorry it is so ugly):

We begin by finding the expected value using iterated expectation

$\begin{align*} E[\mathbf{Z}]&=E[E[\mathbf{Z}|I]] \\ &= E[p\mu_1+(1-p)\mu_2]\\ &= p\mu_1+(1-p)\mu_2 \end{align*}$

Next we note that the group specific covariances can be written as

$\begin{align*} Cov[\mathbf{Z}|I=1] &= E[\mathbf{Z}\mathbf{Z}^T|I=1]-\mu_1\mu_1^T = \mathbf{\Sigma}_1\\ Cov[\mathbf{Z}|I=0] &= E[\mathbf{Z}\mathbf{Z}^T|I=0]-\mu_2\mu_2^T = \mathbf{\Sigma}_2\\ E[\mathbf{Z}\mathbf{Z}^T|I=1] &= \mathbf{\Sigma}_1 + \mu_1\mu_1^T\\ E[\mathbf{Z}\mathbf{Z}^T|I=0] &= \mathbf{\Sigma}_2 + \mu_2\mu_2^T \end{align*} $

Using this same equation for covariance we get

$\begin{align*} Cov[\mathbf{Z}] &= E[\mathbf{Z}\mathbf{Z}^T] - E[\mathbf{Z}]E[\mathbf{Z}]^T\\ &=E[E[\mathbf{Z}\mathbf{Z}^T|I]] - (p\mu_1+(1-p)\mu_2)(p\mu_1+(1-p)\mu_2)^T\\ &= E[p(\mathbf{\Sigma}_1 + \mu_1\mu_1^T)+(1-p)(\mathbf{\Sigma}_2 + \mu_2\mu_2^T)] - (p\mu_1+(1-p)\mu_2)(p\mu_1^T+(1-p)\mu_2^T)\\ &= p\mathbf{\Sigma}_1 + p\mu_1\mu_1^T+(1-p)\mathbf{\Sigma}_2 + (1-p)\mu_2\mu_2^T - (p\mu_1+(1-p)\mu_2)(p\mu_1^T+(1-p)\mu_2^T)\\ &=p\mathbf{\Sigma}_1 + p\mu_1\mu_1^T+(1-p)\mathbf{\Sigma}_2 + (1-p)\mu_2\mu_2^T - (p^2\mu_1\mu_1^T+p(1-p)\mu_1^T\mu_2+p(1-p)\mu_1\mu_2^T+(1-p)^2\mu_2\mu_2^T)\\ &=p\mathbf{\Sigma}_1+(1-p)\mathbf{\Sigma}_2+(p-p^2)(\mu_1\mu_1^T)-p(1-p)\mu_1^T\mu_2-p(1-p)\mu_1\mu_2^T+(p-p^2)(\mu_2\mu_2^T)\\ &=p\mathbf{\Sigma}_1+(1-p)\mathbf{\Sigma}_2+p(1-p)\left[(\mu_1\mu_1^T)-\mu_1^T\mu_2-\mu_1\mu_2^T+(\mu_2\mu_2^T)\right]\\ &= p\mathbf{\Sigma}_1+(1-p)\mathbf{\Sigma}_2+p(1-p)(\mu_1-\mu_2)(\mu_1^T-\mu_2^T)\\ &= p\mathbf{\Sigma}_1+(1-p)\mathbf{\Sigma}_2+p(1-p)(\mu_1-\mu_2)(\mu_1-\mu_2)^T\\ \end{align*}$

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.