From the book "Machine Learning a probabilistic Perspective", I'm reading about finite/infinite mixture models. Particularly at paragraph 25.2.1 it's stated:

The usual representation (of a finite mixture model) is as follows:

$p(x_i|z_i = k, \boldsymbol\theta) = p(x_i|\boldsymbol\theta_k)$

$p(z_i = k| \boldsymbol\pi = \pi_k) = \pi_k$

$p(\boldsymbol\pi|\alpha) =\text{Dir}(\boldsymbol\pi|(\alpha/K)\boldsymbol1_K)$

The form of $p(\boldsymbol\theta_k|\lambda)$ is chosen to be be conjugate to $p(x_i|\boldsymbol\theta_k)$. We can write $p(x_i|\boldsymbol\theta_k)$ as $\boldsymbol{x}_i \sim F(\boldsymbol\theta_{z_i})$ where F is the observation distribution. Similarly, we can write $\boldsymbol\theta_k \sim H(\lambda)$, where H is the prior.

Now this modelling is quite confusing to me. What is the difference between $\boldsymbol\theta_k$ and $\boldsymbol\theta_{z_i}$? What is meant by "Observation distribution"? Can we apply EM algorithm to this model, how?


If you consider a mixture model,$$X\sim\sum_k \pi_k p(x|\theta_k)$$it can be expressed as the marginal of the joint model$$(X,Z)\sim \underbrace{\pi_z}_{p_\pi(z)}\times\underbrace{p(x|\theta_z)}_{p_\theta(x|z)}$$where $Z$ is an integer valued random variable. This implies that$$X|Z=k\sim p(x|\theta_k)$$which can also be written as $$X|Z\sim p(x|\theta_Z)$$The random variable $Z$ is also latent in that (a) it is not observed and (b) it does not necessarily "exist" in the original experiment modelled by the mixture. But the EM algorithm that returns the MLE of the parameters $\pi_k$ and $\theta_k$ takes advantage of this joint representation by iteratively

  1. calculating $\mathbb{E}[Z_i|X_i,\theta,\pi]$
  2. maximising the expected log-likelihood in $(\theta,\pi)$
  • $\begingroup$ Thank you! And what is the difference between $\boldsymbol\theta_k$ and $\boldsymbol\theta_{z_i}$? And what is meant by "We can write > $p(x_i|\boldsymbol\theta_k)$ as $\boldsymbol{x}_i \sim F(\boldsymbol\theta_{z_i})$ where F is the observation distribution"? $\endgroup$ – Tommaso Bendinelli Jan 10 '19 at 12:23
  • $\begingroup$ If $ X|Z=k\sim p(x|\theta_k)$ and $X|Z\sim p(x|\theta_Z)$ are identical, because both represent the random varialbe $X|Z$ why in one case $\theta$ is $\theta_k$ and in the other $\theta_z$ ? $\endgroup$ – Tommaso Bendinelli Jan 10 '19 at 12:42
  • $\begingroup$ I don't get both your point, (i) where have you seen $X|Z = Z|X$, (ii) What you "by is indexed". $\endgroup$ – Tommaso Bendinelli Jan 10 '19 at 12:49
  • $\begingroup$ Also do we know the distribution $F(\theta_{z_i})$? $\endgroup$ – Tommaso Bendinelli Jan 10 '19 at 12:54

What is the difference between $\theta_k$ and $\theta_{z_i}$?

The only difference is the subscript. The value $\theta_{z_i}$ refers to the value $\theta_k$ where $k = z_i$. So the bit which says $\boldsymbol{x} \sim F(\boldsymbol{\theta}_{z_i})$ simply means that $\boldsymbol{x} \sim F(\boldsymbol{\theta}_k)$ where the subscript $k$ is the latent random variable $z_i$.

  • $\begingroup$ Thanks for answering, do we know the distribution $(F(\boldsymbol{\theta_k}))$ or is it unknown? Intuitively I would think that we don't know it, because if we do there is no point in modeling a Gaussian mixture right? $\endgroup$ – Tommaso Bendinelli Jan 10 '19 at 13:29
  • $\begingroup$ It is not specified in the problem, since this is a general description of the finite mixture model, which can take on all sorts of distributions. Usually it would be a distribution with an assumed particular parametric form, but with unknown parameters. $\endgroup$ – Ben Jan 10 '19 at 13:31
  • $\begingroup$ Okey, so just to be sure I understood. $\theta_{z_i}$ are random variable, because $z_i$ is random variable. While $\theta_k$, once sampled from H it's a value. Correct? $\endgroup$ – Tommaso Bendinelli Jan 10 '19 at 13:33
  • $\begingroup$ Yes, but in a mixture model you will usually only observe the $x$s. The rest are unobserved "latent variables" which are effectively parameter values that you infer via Bayes theorem. $\endgroup$ – Ben Jan 10 '19 at 13:36
  • $\begingroup$ Please have a look here stats.stackexchange.com/questions/386681/… $\endgroup$ – Tommaso Bendinelli Jan 11 '19 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.