Confusion in modelling finite mixture model

Question

From the book "Machine Learning a probabilistic Perspective", I'm reading about finite/infinite mixture models. Particularly at paragraph 25.2.1 it's stated:

The usual representation (of a finite mixture model) is as follows:

$p(x_i|z_i = k, \boldsymbol\theta) = p(x_i|\boldsymbol\theta_k)$

$p(z_i = k| \boldsymbol\pi = \pi_k) = \pi_k$

$p(\boldsymbol\pi|\alpha) =\text{Dir}(\boldsymbol\pi|(\alpha/K)\boldsymbol1_K)$

The form of $p(\boldsymbol\theta_k|\lambda)$ is chosen to be be conjugate to $p(x_i|\boldsymbol\theta_k)$. We can write $p(x_i|\boldsymbol\theta_k)$ as $\boldsymbol{x}_i \sim F(\boldsymbol\theta_{z_i})$ where F is the observation distribution. Similarly, we can write $\boldsymbol\theta_k \sim H(\lambda)$, where H is the prior.

Now this modelling is quite confusing to me. What is the difference between $\boldsymbol\theta_k$ and $\boldsymbol\theta_{z_i}$? What is meant by "Observation distribution"? Can we apply EM algorithm to this model, how?

Answer 1

If you consider a mixture model,$$X\sim\sum_k \pi_k p(x|\theta_k)$$it can be expressed as the marginal of the joint model$$(X,Z)\sim \underbrace{\pi_z}_{p_\pi(z)}\times\underbrace{p(x|\theta_z)}_{p_\theta(x|z)}$$where $Z$ is an integer valued random variable. This implies that$$X|Z=k\sim p(x|\theta_k)$$which can also be written as $$X|Z\sim p(x|\theta_Z)$$The random variable $Z$ is also latent in that (a) it is not observed and (b) it does not necessarily "exist" in the original experiment modelled by the mixture. But the EM algorithm that returns the MLE of the parameters $\pi_k$ and $\theta_k$ takes advantage of this joint representation by iteratively

1. calculating $\mathbb{E}[Z_i|X_i,\theta,\pi]$
2. maximising the expected log-likelihood in $(\theta,\pi)$

Answer 2

What is the difference between $\theta_k$ and $\theta_{z_i}$?

The only difference is the subscript. The value $\theta_{z_i}$ refers to the value $\theta_k$ where $k = z_i$. So the bit which says $\boldsymbol{x} \sim F(\boldsymbol{\theta}_{z_i})$ simply means that $\boldsymbol{x} \sim F(\boldsymbol{\theta}_k)$ where the subscript $k$ is the latent random variable $z_i$.