# Use Expectation-Maximization algorithm for obtaining maximal likelihood estimator

For $$X = {(Z_{i}, Y_{i}) : i = 1, ... ,n}$$, consider the model:

$$Y_{i} = \beta_{1} + \beta_{2}Z_{i} + \epsilon_{i}$$ where $$\epsilon_{1}, ... ,\epsilon_{n}$$ are i.i.d $$N(0,\sigma^2)$$, $$Z_{i},...Z_{i}$$ are i.i.d $$N(\mu_{1},\sigma_{1}^2)$$ and independent of $$\epsilon_{1}, ... ,\epsilon_{n}$$.

For $$1\le i\le m$$, we observe both $$Z_i$$ and $$Y_i$$, and for $$m + 1 \le i\le n$$, we observe only $$Y_i$$.

I try to complete the E- and M-steps of the EM algorithm for estimating $$\theta=(\mu_{1},\beta_{1},\sigma_{1}^2,\sigma^2,\beta_{2})$$

E-step: $$Q(\theta; \theta_{old}) = E [\ell(θ; X , Y) | X , \theta_{old}]$$

M-step: $$\theta_{new} := \max\limits_{\theta} Q(\theta; \theta_{old})$$.

still stuggle how to apply these step to solve the problem

• Have you lost interest in the question? Mar 23, 2020 at 15:17

You could express the complete log-likelihood as $$\ell(\theta) = -n\log\sigma_1-\frac{1}{2}\sum_{i=1}^n \sigma_1^{-2}(z_i-\mu_1)^2 -n\log\sigma^2-\frac{1}{2}\sum_{i=1}^n \sigma^{-2}(y_i-\beta_1-\beta_2 z_i)^2$$ to derive $$\mathbb E_{\theta^\text{old}}[\ell(\theta)|X^\text{obs}]$$as \begin{align}&-n\log\sigma_1-\frac{1}{2}\sum_{i=1}^m \sigma_1^{-2}(z_i-\mu_1)^2\\&-\frac{\sigma_1^{-2}}{2}\sum_{i=m+1}^n \mathbb E_{\theta^\text{old}}[(Z_i-\mu_1)^2|y_i]\\ &-n\log\sigma^2-\frac{1}{2}\sum_{i=1}^m \sigma^{-2}(y_i-\beta_1-\beta_2 z_i)^2\\ &-\frac{\sigma^{-2}}{2}\sum_{i=m+1}^n \mathbb E_{\theta^\text{old}}[(y_i-\beta_1-\beta_2 Z_i)^2|y_i] \end{align} and then use the conditional distribution of $$Z_i$$ given $$Y_i=y_i$$ and $$\theta^\text{old}$$: $$f(z_i|y_i,\theta)\propto\exp\{-\sigma^{-2}(y_i-\beta_1-\beta_2z_i)^2/2\}\times\exp\{-\sigma_1^{-2}(z_i-\mu_1)^2/2\}$$ which means $$Z_i|Y_i,\theta\sim\mathcal N\left(\{\sigma^-2\beta_2^2+\sigma_1^{-1}\}^{-1}\{\sigma^-2\beta_2[y_i-\beta_1]\},\{\sigma^-2\beta_2^2+\sigma_1^{-1}\}^{-1}\right)$$ from which $$\mathbb E_{\theta^\text{old}}[(Z_i-\mu_1)^2|y_i]\qquad\text{and}\qquad\mathbb E_{\theta^\text{old}}[(y_i-\beta_1-\beta_2 Z_i)^2|y_i]$$ are easily derived.