I'm working through chapter 9 in Bishop (Mixture models and EM) and I'm stuck on equation 9.29. For those without the book:

Bishop states that the log likelihood for a latent variable model with

  1. data, $X$
  2. latent variables, $Z$
  3. latent model parameters $\theta$

is given by:

$$ ln \; p(\textbf{X}| \boldsymbol{\theta}) = ln \; \left\{ \sum_{\textbf{Z}} p(\textbf{X,Z}|\boldsymbol{\theta} ) \right\} \: \: (1) $$

Shouldn't the log likelihood be over the data as well as the latent variables? Every example I've seen of a likelihood is over the data. If this is the case shouldn't the likelihood be:

$$ p(\textbf{X}| \boldsymbol{\theta}) = \prod_{ \textbf{x} \in \textbf{X}} \sum_{\textbf{Z}} p(\textbf{x,Z}|\boldsymbol{\theta} ) \: \: (2) $$

(multiplying over the data)

$$ ln \: p(\textbf{X}| \boldsymbol{\theta}) = \sum_{ \textbf{x} \in \textbf{X} } ln \left\{ \sum_{\textbf{Z}} p(\textbf{x,Z}|\boldsymbol{\theta} ) \right\} \: \: (3) $$

where $\textbf{x}$ is one observation in $\textbf{X}$

It strikes me that (1) is the equivalent of equation $9.7$ in Bishop (p430)

$$ p(\textbf{x}) = \sum_{k=1}^K \pi_k {N}(\textbf{x}| \boldsymbol{\mu} , \boldsymbol{\sigma}) \: \: (4)$$

Which is not described as a likelihood. When (4) is turned into a likelihood later on (equation 9.14), it is given as:

$$ ln\; p(X|\pi,\mu,\Sigma) = \sum_{n=1}^{N} ln \left\{ \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) \right\} \: \: (5) $$

where Bishop sums over the latent variable $Z$ and the observations $x$.Which looks an awful lot like my attempt at the log likelihood (3)

My questions:

  1. Surely if the progression from (4) to (5) holds shouldn't the log likelihood be (3) rather than (1) ?

  2. Did bishop just miss out the sum over the data?


(1) is the general formula obtained using marginalisation. In (2), you assume iid data samples, so the joint distribution transforms into multiplication of marginals, e.g. $$p(\mathbf{X})=p(x_1,x_2)=p(x_1)p(x_2)$$ In the book, the sentence before equation 9.14, equation (5) in your question, states that the dataset is assumed to be iid.

  • $\begingroup$ Thanks again @gunes. Two questions: 1. What has been marginalised over in equation (1)? 2. Does equation (1) NOT assume IID samples then? $\endgroup$ – RNs_Ghost Mar 25 '20 at 23:17
  • 2
    $\begingroup$ 1. The joint $p(X,Z|\theta)$ is marginalised over $Z$ to obtain $p(X|\theta)$ 2. (1) doesn't assume iid samples, it's generic, i.e. $p(X)\neq \prod p(x)$ always. $\endgroup$ – gunes Mar 25 '20 at 23:19
  • $\begingroup$ Right! I think I understand. Should I always assume, when I come across things in the future data is NOT IID unless it says so? (Particularly if I see $p(X)$ rather than $ \prod p(x)$ (Maybe this is a silly question...) $\endgroup$ – RNs_Ghost Mar 25 '20 at 23:24
  • 2
    $\begingroup$ Yes, iid assumption is typical but extra. $\endgroup$ – gunes Mar 25 '20 at 23:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.