EM algorithm for Bivariate Normal

Question

Consider a random sample $X_i = (U_i,V_i)$ where $i=1,2,...,n$ from a bivariate normal population with mean $(\mu_1,\mu_2)$ and variances $(\sigma_1 ^2, \sigma_2 ^2)$ and correlation $\rho$. Let's consider random variables $C_1,...,C_n$ which take the values $\{0,1,2\}$ and are independent of $X_i$. We observe each $C_i$ Now, if $C_i=0$, we observe $X_i$; if $C_i=1$, we observe only $U_i$; and finally if $C_i=2$, we observe $V_i$. I want to develop an EM algorithm to estimate the parameters of the distribution of $X_i$ given data.

My approach:

Let us define $Z_i = X_i$ if $C_i=0$ say, with probability ($\pi_1$), $Z_i=U_i$ if $C_i=1$ with probability $\pi_2$ and $Z_i=V_i$ if $C_i=2$ with probability $\pi_3$. Since, we observe both $Z_i,C_i$, the observed data likelihood is: $$L = \prod_{i=1}^{n} (\pi_1 f_{X}(x_i) + \pi_2 f_{U}(u_i) +\pi_3 f_{V}(v_i))$$.

In general, for EM algorithm we are required to find for the E-step: $E[\text{unobserved data}|\text{observed data}]$, but I am a bit confused with guessing what that should be in this case. In this case, if we observe $C_i=0$ then both $U_i,V_i$ are observed. However, if we observe $C_i=1$, then only $V_i$ is unobserved and for $C_i=2$, then $U_i$ is unobserved. So, does that mean our E step should deal with $E[V_i|U_i]$ and $E[U_i|V_i]$?

Can anyone help me out with this problem?

Answer 1

Denoting the Normal parameters by $\theta$, the observed likelihood is $$\pi_0^{n_0}\pi_1^{n_1}\pi_2^{n_2}\prod_{i:\,c_i=0}f_X(u_i,v_i;\theta) \prod_{i:\,c_i=1}f_U(u_i;\theta)\prod_{i:\,c_i=2}f_V(v_i;\theta)$$ with each term being in closed (Normal) form. Hence this likelihood can be optimised in $(\theta,\pi)$ and does not require an EM algorithm.

If EM is implemented as a toy problem, the completed likelihood is $$\pi_0^{n_0}\pi_1^{n_1}\pi_2^{n_2}\prod_{i:\,c_i=0}f_X(u_i,v_i;\theta) \prod_{i:\,c_i=1}f_X(u_i,V_i;\theta) \prod_{i:\,c_i=2}f_X(U_i,v_i;\theta)$$ where the missing / latent variables are upper cases. Removing the $\pi_j$'s in front which do not require at all EM to be optimised as $$\hat\pi_i=n_i/n,$$ the E function (on $\theta$) is $$\begin{align}\text{E}[\theta'|\theta,\mathcal D]&= \sum_{i:\,c_i=0}\log f_X(u_i,v_i;\theta')+ \sum_{i:\,c_i=1}\mathbb E_\theta[\log f_X(u_i,V_i;\theta')|U_i=u_i]\\ &\qquad+ \sum_{i:\,c_i=2}\mathbb E_\theta[\log f_X(U_i,v_i;\theta')|V_i=v_i]\\ &=\sum_{i:\,c_i=0}\log f_X(u_i,v_i;\theta')\\ &\qquad+\sum_{i:\,c_i=1}\mathbb \log f_U(u_i;\theta')+ \sum_{i:\,c_i=1}\mathbb E_\theta[\log f_{V|U}(V_i;\theta',U_i=u_i])|U_i=u_i]\\ &\qquad+\sum_{i:\,c_i=2}\log f_V(v_i;\theta')+ \sum_{i:\,c_i=2}\mathbb E_\theta[\log f_{U|V}(U_i;\theta',V_i=v_i)|V_i=v_i]\end{align}$$ [where all density terms are normal] to be optimised in $\theta'$ (M-step).

Interestingly, albeit generically, this E-function also writes as $$\overbrace{\sum_{i:\,c_i=0}\log f_X(u_i,v_i;\theta')+\sum_{i:\,c_i=1}\mathbb \log f_U(u_i;\theta')+\sum_{i:\,c_i=2}\log f_V(v_i;\theta')}^\text{observed log-likelihood}\\ \left. \begin{align}&+\sum_{i:\,c_i=1}\mathbb E_\theta[\log f_{V|U}(V_i;\theta',U_i=u_i])|U_i=u_i]\\ &+\sum_{i:\,c_i=2}\mathbb E_\theta[\log f_{U|V}(U_i;\theta',V_i=v_i)|V_i=v_i] \end{align}\right.$$ which means that close to (EM) convergence the expected part should not contribute significantly. When reaching a fixed point, i.e., when $\theta=\theta'$, $$\mathbb E_\theta[\log f_{V|U}(V;\theta',U=u_i])|U=u_i]=-\frac{1}{2}$$