How does the EM algorithm operate when group label may be missing?

Question

I have a set of data where the group label for a bunch of the data is missing. I know that there are 10 groups (integer 1 to 10):

    label   x1      x2    b1
1      NA  2.311 6.633     1
2      NA  4.198 2.483     0
3       2  2.655 2.960     1
4      NA  4.315 2.732     0
5       1  2.631 3.427     1
6      NA  1.794 3.499     0

x1 and x2 follow normal distribution with mean dependent of the class label and known variance, b1 follow binomial distribution with p that is also dependent of the class label. The probability of each case being assigned to class $i$ is $p_i$ which sum to 1 for all classes.

I know that if the classes are all known, we can just calculate the class-specific log likelihood, but when some of them are unknown, how should I proceed it?

Answer 1

Your model can be formalised as follows: $$\eqalign{ X_{1i}|Z_i=k&\sim\mathcal{N}(\mu_{1k},\sigma_1^2)\\ X_{2i}|Z_i=k&\sim\mathcal{N}(\mu_{2k},\sigma_2^2)\\ B_i|Z_i=k &\sim \mathcal{B}(q_k)\\ Z_i&\sim\mathcal{M}(p_1,\ldots,p_K) }$$ This means the complete likelihood is $$ \prod_{i=1}^n p_{z_i} q_{z_i}^{b_i}(1-q_{z_i})^{1-b_i}\prod_{j=1}^2\varphi(x_{ji};\mu_{jz_i},\sigma_j) $$ and the log complete likelihood is $$ \sum _{i=1}^n \sum_{k=1}^K z_{ik} \left\{\log(p_k)+b_i\log(q_k) +(1-b_i)\log(1-q_k)+\sum_{j=1}^2 \log \varphi(x_{ji};\mu_{jk},\sigma_j) \right\} $$ where $z_{ik}$ denotes the indicator $\mathbb{I}_k(z_i)$.

The EM algorithm starts from this complete likelihood to compute an expectation (E step) $$\eqalign{ &\mathbb{E}_{\theta_0}\left[\sum _{i=1}^n \sum_{k=1}^K z_{ik} \left\{\log(p_k)+b_i\log(q_k) +(1-b_i)\log(1-q_k)\right.\right.\\ &\qquad\left.\left.+\sum_{j=1}^2 \log \varphi(x_{ji};\mu_{jk},\sigma_j) \right\}\Big|\mathcal{D}\right] }$$ where $\mathcal{D}$ denotes the observed data (which includes the observed $z_i$'s) and $\theta_0$ the current value of the parameter. That is, $$\eqalign{ Q(\theta_0,\theta)&=\sum _{i=1}^n \sum_{k=1}^K \mathbb{E}_{\theta_0}[z_{ik}\big|\mathcal{D}] \Big\{\log(p_k)+b_i\log(q_k) +(1-b_i)\log(1-q_k)\\ &\qquad\left.+\sum_{j=1}^2 \log \varphi(x_{ji};\mu_{jk},\sigma_j) \right\} }$$ EM follows by maximising (M step) the above wrt to all the unknown parameters $$\theta_1=\arg\max_\theta Q(\theta_0,\theta)$$