Question about the latent variable in EM algorithm

Question

In mixture models, Expectation maximization algorithm (EM) is a commonly used method to estimate the model parameters. Suppose that I have bivariate mixture model with two mixture components, with mixture weights, $\pi_1$ and $\pi_2$, respectively. EM introduces a $z$ variable which takes 1 if the point is come from the first mixture component, and takes 0 otherwise. These variables are assumed to be i.i.d and are distributed from multinomial ($\pi_1$, $\pi_2$). Is that correct? If yes, how they can have only {0,1} values and in the sometimes they have multinomial distribution?

Here is the paragraph from the source: here is the link

3.2. EM algorithm

In this section, we describe the EM algorithm (Dempster et al., 1977) to obtain the estimates for the parameters θ in a mixture of M-component D-vine densities, given the data set and the number of components M. The determination of M will be discussed later in Section 3.3. Assume that N observations, say $x_k = (x_{k,1}, . . . , x_{k,N} )$ where $k = 1, . . . , d$, drawn randomly from a $M$-component.

Let us denote latent variables $z_{n} = (z_{n1}, . . . , z_{nm}, . . . , z_{nM})$, where $z_{nm} = 1$ if $x_{n}$ comes from the $m$-th component and $z_{nm}$ = 0 otherwise. Assume that $z_{n}$ is independent and identically distributed from a multinomial distribution, that is, $z_{n} ∼ Mult(M, π = (π_{1}, . . . , π_{M}))$.

Any help, please?

Answer 1

There is a lot of confusion in the question, confusion that could be reduced by looking at a textbook on the paper, or even the original 1977 paper by Dempster, Laird and Rubin.

Here is an excerpt of our book, Introducing Monte Carlo Methods with R, followed by my answer:

Assume that we observe $X_1, \ldots, X_n$, jointly distributed from $g({\mathbf x}|\theta)$ that satisfies $$ g({\mathbf x}|\theta)=\int_{\cal Z} f({\mathbf x}, {\mathbf z}|\theta)\, \text{d}{\mathbf z}, $$ and that we want to compute $\hat{\theta} = \arg\max L(\theta|{\mathbf x})= \arg\max g({\mathbf x}|\theta)$. Since the augmented data is ${\mathbf z}$, where $({\mathbf X}, {\mathbf Z}) \sim f({\mathbf x},{\mathbf z}| \theta)$ the conditional distribution of the missing data ${\mathbf Z}$ given the observed data ${\mathbf x}$ is $$ k({\mathbf z}| \theta, {\mathbf x}) = f({\mathbf x}, {\mathbf z}|\theta)\big/g({\mathbf x}|\theta)\,. $$ Taking the logarithm of this expression leads to the following relationship between the complete-data likelihood $L^c(\theta|{\mathbf x}, {\mathbf z})$ and the observed-data likelihood $L(\theta|{\mathbf x})$. For any value $\theta_0$, $$ \log L(\theta|{\mathbf x})= \mathbb{E}_{\theta_0}[\log L^c(\theta|{\mathbf x},{\mathbf Z})] -\mathbb{E}_{\theta_0}[\log k({\mathbf Z}| \theta, {\mathbf x})],\qquad(1) $$where the expectation is with respect to $k({\mathbf z}| \theta_0, {\mathbf x})$. In the EM algorithm, while we aim at maximizing $\log L(\theta|{\mathbf x})$, only the first term on the right side of (1) will be considered.

Denoting$$ Q(\theta |\theta_0, {\mathbf x}) = \mathbb{E}_{\theta_0} [\log L^c(\theta|{\mathbf x},{\mathbf Z})], $$ the EM algorithm indeed proceeds iteratively by maximizing $Q(\theta |\theta_0, {\mathbf x})$ at each iteration and, if $\hat{\theta}_{(1)}$ is the value of $\theta$ maximizing $Q(\theta |\theta_0, {\mathbf x})$, by replacing $\theta_0$ by the updated value $\hat{\theta}_{(1)}$. In this manner, a sequence of estimators $\{\hat{\theta}_{(j)}\}_j$ is obtained, where $\hat{\theta}_{(j)}$ is defined as the value of $\theta$ maximizing $Q(\theta |\hat{\theta}_{(j-1)}, {\mathbf x})$; that is,$$ Q(\hat{\theta}_{(j)} |\hat{\theta}_{(j-1)}, {\mathbf x}) = \max_{\theta}\,Q(\theta |\hat{\theta}_{(j-1)}, {\mathbf x}).$$This iterative scheme thus contains both an expectation step and a maximization step, giving the algorithm its name.

EM Algorithm Pick a starting value $\hat{\theta}_{(0)}$

Repeat

1. Compute the E-step $$ Q(\theta|\hat{\theta}_{(m)}, {\mathbf x}) =\mathbb{E}_{\hat{\theta}_{(m)}} [\log L^c(\theta|{\mathbf x}, {\mathbf Z})]\,, $$ where the expectation is with respect to $k({\mathbf z}|\hat{\theta}_{(m)},{\mathbf x})$ and set $m=0$.

2. Maximize $Q(\theta|\hat{\theta}_{(m)}, {\mathbf x})$ in $\theta$ and take the M-step $$ \hat\theta_{(m+1)}=\arg\max_\theta \; Q(\theta|\hat{\theta}_{(m)}, {\mathbf x}) $$ and set $m=m+1$

until a fixed point is reached; i.e., $\hat\theta_{(m+1)}=\hat{\theta}_{(m)}$.

For the normal mixture, using the missing data structure exhibited in previously leads to an objective function equal to $$ Q(\theta^\prime|\theta,\mathbf{x}) = -\frac{1}{2}\,\sum_{i=1}^n \mathbb{E}_\theta\left[\left. Z_i (x_i-\mu_1)^2 + (1-Z_i) (x_i-\mu_2)^2 \right| \mathbf{x} \right]. $$ Solving the M-step then provides the closed-form expressions $$ \mu_1^\prime = \mathbb{E}_\theta\left[ \sum_{i=1}^n Z_i x_i |\mathbf{x} \right] \bigg/ \mathbb{E}_\theta\left[ \sum_{i=1}^n Z_i| \mathbf{x} \right] $$ and $$ \mu_1^\prime = \mathbb{E}_\theta\left[ \sum_{i=1}^n (1-Z_i) x_i |\mathbf{x} \right] \bigg/ \mathbb{E}_\theta\left[ \sum_{i=1}^n (1-Z_i)| \mathbf{x} \right]. $$ Since $$ \mathbb{E}_\theta\left[Z_i|\mathbf{x} \right]=\frac{\varphi(x_i-\mu_1)}{ \varphi(x_i-\mu_1)+3\varphi(x_i-\mu_2)}\,, $$ the EM algorithm can easily be implemented in this setting.

Whatever the mixture involved, the latent variables $Z_i$ are Multinomial $\mathcal{M}_M(1;\pi_1,\ldots,\pi_M)$ which means only one component of the vector $Z_i$ is equal to one and all of the $M-1$ others are zero. (Note the difference with the question in the notations: the original notation $\mathcal{M}(M;\pi_1,\ldots,\pi_M)$ fails to indicate how many draws are taken, that is, what is the sum of the components of $Z_i$.).When $k=2$ as in the above excerpt, $Z_i$ is an integer in $\{0,1\}$. There may be a confusion between a Multinomial distribution and the property of a distribution (like some mixtures) to be multimodal. The $Z_i$ do not have a multimodal distribution, taking only two values, even conditional on the $X_i$'s, while the $X_i$'s may, at least unconditionally.

Answer 2

If I correctly read between the lines, your question is about the difference between the distribution of $[z]$ (i.e., the prior distribution of the latent variable), and the distribution of $[z \mid y]$ (i.e., the posterior distribution of the latent variable given the data $y$).

The prior probabilities are the same for all subjects, i.e., $\pi$. However, the posterior probabilities will be different per subject because each subject will his/her own probability of belonging in the first component (i.e., the component for which $z_i = 1$) depending on their data $y_i$. Hence, if you do the plot of the posterior probabilities, it can be multimodal.