I am reading the materials on the EM algorithm, and I am a bit confused about the example provided on the material I am currently reading. The example is considered a classical missing data problem solving by EM algorithm. Here is the content of the material I am reading :

197 animals are distributed into four categories $$ \left(x_{1}, x_{2}, x_{3}, x_{4}\right)=(125,18,20,34) $$ and modeled with the multinomial distribution $$ \text { multinomial }\left(n ; \frac{1}{2}+\frac{\theta}{4}, \frac{1}{4}(1-\theta), \frac{1}{4}(1-\theta), \frac{\theta}{4}\right) $$ Estimation is easier if the $x_{1}$ cell is split into two cells, so we create the augmented model $$ \left(z_{1}, z_{2}, x_{2}, x_{3}, x_{4}\right) \sim \text { multinomial }\left(n ; \frac{1}{2}, \frac{\theta}{4}, \frac{1}{4}(1-\theta), \frac{1}{4}(1-\theta), \frac{\theta}{4}\right) $$ with $x_{1}=z_{1}+z_{2}$

In the introduction section of the EM algorithm, the setting is as follow:

Consider a sample of $n$ items, where $n_{1}$ of the items are observed while $n_{2}=n-n_{1}$ items are not observable Let $Z_{j}$ 's and $X_{i}$ 's be mutually independent and $$ \begin{array}{ll} X_{1}, \ldots, X_{n_{1}} \sim g(x \mid \theta) & -\text { Observed Data } \\ Z_{1}, \ldots, Z_{n_{2}} \sim f(z \mid \theta) & -\text { Unobserved Data } \end{array} $$ $(\boldsymbol{X}, \boldsymbol{Z})$ is called the complete data, $\boldsymbol{Z}$ the augmented data, and $\boldsymbol{X}$ the incomplete data.

Here, there is a clear cut between augmented data and incomplete data. However, here I am rather confused: it seems $(x_1,x_2,x_3,x_4)$ will be the incomplete data while $(z_1,z_2,x_2,x_3,x_4)$ will be consider the complete data. I have no previous exposure to the concept of augmented data and I am very confused why simply introducing two other random variables would work.

Also, I know the pdf for multinomial distribution with four outcomes have expression $$ f\left(x_{1}, x_{2}, x_{3}, x_{4}\right)=\frac{n !}{x_{1} ! x_{2} ! x_{3} ! x_{4} !} \pi_{1}^{x_{1}} \pi_{2}^{x_{2}} \pi_{3}^{x_{3}} \pi_{4}^{x_{4}} $$

if we substitute $x_1$ with $z_1+z_2$, we have $(\pi_1+c)^{z_1}(\pi_1+c)^{z_2}$ but not $\pi_1^{z_1}c^{z_2}$, also, the factorial part of $x_1!=(z_1+z_2)!\neq z_1!z_2!$ How the substitution actually work here? What makes this substitution valid?


1 Answer 1


Here $g(x_1,x_2,x_3,x_4)$ is the distribution associated with observed data, whereas $f(z_1,z_2,x_2,x_3,x_4)$ is the distribution associated with the complete data. So, we are not substituting the variables $z_1,z_2$ into $g$. Rather, these are two different distributions and there is no reason why $g(x_1,x_2,x_3,x_4) = f(z_1,z_2,x_2,x_3,x_4), \text{where,} \quad x_1 = z_1 + z_2$ should hold in general.

The only relation between $g$ and $f$ is that $g(x_1,x_2,x_3,x_4) = \sum_{z_1,z_2}f(z_1,z_2,x_2,x_3,x_4)$ where the summation is over all possible $z_1,z_2 \geq 0$ such that $x_1 = z_1 + z_2$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.