From the book "Machine Learning a probabilistic Perspective", I'm reading about finite/infinite mixture models. Particularly at paragraph 25.2.1 it's stated:

The usual representation (of a finite mixture model) is as follows:

$p(x_i|z_i = k, \boldsymbol\theta) = p(x_i|\boldsymbol\theta_k)$

$p(z_i = k| \boldsymbol\pi = \pi_k) = \pi_k$

$p(\boldsymbol\pi|\alpha) =\text{Dir}(\boldsymbol\pi|(\alpha/K)\boldsymbol1_K)$

The form of $p(\boldsymbol\theta_k|\lambda)$ is chosen to be be conjugate to $p(x_i|\boldsymbol\theta_k)$. We can write $p(x_i|\boldsymbol\theta_k)$ as $\boldsymbol{x}_i \sim F(\boldsymbol\theta_{z_i})$ where F is the observation distribution. Similarly, we can write $\boldsymbol\theta_k \sim H(\lambda)$, where H is the prior.

Now this modelling is quite confusing to me. What is the difference between $\boldsymbol\theta_k$ and $\boldsymbol\theta_{z_i}$? What is meant by "Observation distribution"? Can we apply EM algorithm to this model, how?


2 Answers 2


If you consider a mixture model,$$X\sim\sum_k \pi_k p(x|\theta_k)$$it can be expressed as the marginal of the joint model$$(X,Z)\sim \underbrace{\pi_z}_{p_\pi(z)}\times\underbrace{p(x|\theta_z)}_{p_\theta(x|z)}$$where $Z$ is an integer valued random variable. This implies that$$X|Z=k\sim p(x|\theta_k)$$which can also be written as $$X|Z\sim p(x|\theta_Z)$$The random variable $Z$ is also latent in that (a) it is not observed and (b) it does not necessarily "exist" in the original experiment modelled by the mixture. But the EM algorithm that returns the MLE of the parameters $\pi_k$ and $\theta_k$ takes advantage of this joint representation by iteratively

  1. calculating $\mathbb{E}[Z_i|X_i,\theta,\pi]$
  2. maximising the expected log-likelihood in $(\theta,\pi)$
  • $\begingroup$ Thank you! And what is the difference between $\boldsymbol\theta_k$ and $\boldsymbol\theta_{z_i}$? And what is meant by "We can write > $p(x_i|\boldsymbol\theta_k)$ as $\boldsymbol{x}_i \sim F(\boldsymbol\theta_{z_i})$ where F is the observation distribution"? $\endgroup$ Jan 10, 2019 at 12:23
  • $\begingroup$ If $ X|Z=k\sim p(x|\theta_k)$ and $X|Z\sim p(x|\theta_Z)$ are identical, because both represent the random varialbe $X|Z$ why in one case $\theta$ is $\theta_k$ and in the other $\theta_z$ ? $\endgroup$ Jan 10, 2019 at 12:42
  • $\begingroup$ I don't get both your point, (i) where have you seen $X|Z = Z|X$, (ii) What you "by is indexed". $\endgroup$ Jan 10, 2019 at 12:49
  • $\begingroup$ Also do we know the distribution $F(\theta_{z_i})$? $\endgroup$ Jan 10, 2019 at 12:54

What is the difference between $\theta_k$ and $\theta_{z_i}$?

The only difference is the subscript. The value $\theta_{z_i}$ refers to the value $\theta_k$ where $k = z_i$. So the bit which says $\boldsymbol{x} \sim F(\boldsymbol{\theta}_{z_i})$ simply means that $\boldsymbol{x} \sim F(\boldsymbol{\theta}_k)$ where the subscript $k$ is the latent random variable $z_i$.

  • $\begingroup$ Thanks for answering, do we know the distribution $(F(\boldsymbol{\theta_k}))$ or is it unknown? Intuitively I would think that we don't know it, because if we do there is no point in modeling a Gaussian mixture right? $\endgroup$ Jan 10, 2019 at 13:29
  • $\begingroup$ It is not specified in the problem, since this is a general description of the finite mixture model, which can take on all sorts of distributions. Usually it would be a distribution with an assumed particular parametric form, but with unknown parameters. $\endgroup$
    – Ben
    Jan 10, 2019 at 13:31
  • $\begingroup$ Okey, so just to be sure I understood. $\theta_{z_i}$ are random variable, because $z_i$ is random variable. While $\theta_k$, once sampled from H it's a value. Correct? $\endgroup$ Jan 10, 2019 at 13:33
  • $\begingroup$ Yes, but in a mixture model you will usually only observe the $x$s. The rest are unobserved "latent variables" which are effectively parameter values that you infer via Bayes theorem. $\endgroup$
    – Ben
    Jan 10, 2019 at 13:36
  • $\begingroup$ Please have a look here stats.stackexchange.com/questions/386681/… $\endgroup$ Jan 11, 2019 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.