EM algorithm for Bivariate Normal

Consider a random sample $$X_i = (U_i,V_i)$$ where $$i=1,2,...,n$$ from a bivariate normal population with mean $$(\mu_1,\mu_2)$$ and variances $$(\sigma_1 ^2, \sigma_2 ^2)$$ and correlation $$\rho$$. Let's consider random variables $$C_1,...,C_n$$ which take the values $$\{0,1,2\}$$ and are independent of $$X_i$$. We observe each $$C_i$$ Now, if $$C_i=0$$, we observe $$X_i$$; if $$C_i=1$$, we observe only $$U_i$$; and finally if $$C_i=2$$, we observe $$V_i$$. I want to develop an EM algorithm to estimate the parameters of the distribution of $$X_i$$ given data.

My approach:

Let us define $$Z_i = X_i$$ if $$C_i=0$$ say, with probability ($$\pi_1$$), $$Z_i=U_i$$ if $$C_i=1$$ with probability $$\pi_2$$ and $$Z_i=V_i$$ if $$C_i=2$$ with probability $$\pi_3$$. Since, we observe both $$Z_i,C_i$$, the observed data likelihood is: $$L = \prod_{i=1}^{n} (\pi_1 f_{X}(x_i) + \pi_2 f_{U}(u_i) +\pi_3 f_{V}(v_i))$$.

In general, for EM algorithm we are required to find for the E-step: $$E[\text{unobserved data}|\text{observed data}]$$, but I am a bit confused with guessing what that should be in this case. In this case, if we observe $$C_i=0$$ then both $$U_i,V_i$$ are observed. However, if we observe $$C_i=1$$, then only $$V_i$$ is unobserved and for $$C_i=2$$, then $$U_i$$ is unobserved. So, does that mean our E step should deal with $$E[V_i|U_i]$$ and $$E[U_i|V_i]$$?

Can anyone help me out with this problem?

• I couldn't differentiate between the complete data likelihood and the observed data likelihood. Can you elaborate more on this? Commented Nov 4, 2022 at 18:27
• Yes, $C_i$'s are indeed observed Commented Nov 4, 2022 at 18:42

Denoting the Normal parameters by $$\theta$$, the observed likelihood is $$\pi_0^{n_0}\pi_1^{n_1}\pi_2^{n_2}\prod_{i:\,c_i=0}f_X(u_i,v_i;\theta) \prod_{i:\,c_i=1}f_U(u_i;\theta)\prod_{i:\,c_i=2}f_V(v_i;\theta)$$ with each term being in closed (Normal) form. Hence this likelihood can be optimised in $$(\theta,\pi)$$ and does not require an EM algorithm.
If EM is implemented as a toy problem, the completed likelihood is $$\pi_0^{n_0}\pi_1^{n_1}\pi_2^{n_2}\prod_{i:\,c_i=0}f_X(u_i,v_i;\theta) \prod_{i:\,c_i=1}f_X(u_i,V_i;\theta) \prod_{i:\,c_i=2}f_X(U_i,v_i;\theta)$$ where the missing / latent variables are upper cases. Removing the $$\pi_j$$'s in front which do not require at all EM to be optimised as $$\hat\pi_i=n_i/n,$$ the E function (on $$\theta$$) is \begin{align}\text{E}[\theta'|\theta,\mathcal D]&= \sum_{i:\,c_i=0}\log f_X(u_i,v_i;\theta')+ \sum_{i:\,c_i=1}\mathbb E_\theta[\log f_X(u_i,V_i;\theta')|U_i=u_i]\\ &\qquad+ \sum_{i:\,c_i=2}\mathbb E_\theta[\log f_X(U_i,v_i;\theta')|V_i=v_i]\\ &=\sum_{i:\,c_i=0}\log f_X(u_i,v_i;\theta')\\ &\qquad+\sum_{i:\,c_i=1}\mathbb \log f_U(u_i;\theta')+ \sum_{i:\,c_i=1}\mathbb E_\theta[\log f_{V|U}(V_i;\theta',U_i=u_i])|U_i=u_i]\\ &\qquad+\sum_{i:\,c_i=2}\log f_V(v_i;\theta')+ \sum_{i:\,c_i=2}\mathbb E_\theta[\log f_{U|V}(U_i;\theta',V_i=v_i)|V_i=v_i]\end{align} [where all density terms are normal] to be optimised in $$\theta'$$ (M-step).
Interestingly, albeit generically, this E-function also writes as \overbrace{\sum_{i:\,c_i=0}\log f_X(u_i,v_i;\theta')+\sum_{i:\,c_i=1}\mathbb \log f_U(u_i;\theta')+\sum_{i:\,c_i=2}\log f_V(v_i;\theta')}^\text{observed log-likelihood}\\ \left. \begin{align}&+\sum_{i:\,c_i=1}\mathbb E_\theta[\log f_{V|U}(V_i;\theta',U_i=u_i])|U_i=u_i]\\ &+\sum_{i:\,c_i=2}\mathbb E_\theta[\log f_{U|V}(U_i;\theta',V_i=v_i)|V_i=v_i] \end{align}\right. which means that close to (EM) convergence the expected part should not contribute significantly. When reaching a fixed point, i.e., when $$\theta=\theta'$$, $$\mathbb E_\theta[\log f_{V|U}(V;\theta',U=u_i])|U=u_i]=-\frac{1}{2}$$
• I am confused that if $C_i$'s are observed, then is this also the complete data likelihood? Commented Nov 4, 2022 at 19:03
• Appologies, I just realised that my use of upper case $C_i$'s while they are observed may have been the source for your confusion. Commented Nov 5, 2022 at 14:13