# Derivation of EM in bishop

I'm working through chapter 9 in Bishop (Mixture models and EM) and I'm stuck on equation 9.29. For those without the book:

Bishop states that the log likelihood for a latent variable model with

1. data, $$X$$
2. latent variables, $$Z$$
3. latent model parameters $$\theta$$

is given by:

$$ln \; p(\textbf{X}| \boldsymbol{\theta}) = ln \; \left\{ \sum_{\textbf{Z}} p(\textbf{X,Z}|\boldsymbol{\theta} ) \right\} \: \: (1)$$

Shouldn't the log likelihood be over the data as well as the latent variables? Every example I've seen of a likelihood is over the data. If this is the case shouldn't the likelihood be:

$$p(\textbf{X}| \boldsymbol{\theta}) = \prod_{ \textbf{x} \in \textbf{X}} \sum_{\textbf{Z}} p(\textbf{x,Z}|\boldsymbol{\theta} ) \: \: (2)$$

(multiplying over the data)

$$ln \: p(\textbf{X}| \boldsymbol{\theta}) = \sum_{ \textbf{x} \in \textbf{X} } ln \left\{ \sum_{\textbf{Z}} p(\textbf{x,Z}|\boldsymbol{\theta} ) \right\} \: \: (3)$$

where $$\textbf{x}$$ is one observation in $$\textbf{X}$$

It strikes me that (1) is the equivalent of equation $$9.7$$ in Bishop (p430)

$$p(\textbf{x}) = \sum_{k=1}^K \pi_k {N}(\textbf{x}| \boldsymbol{\mu} , \boldsymbol{\sigma}) \: \: (4)$$

Which is not described as a likelihood. When (4) is turned into a likelihood later on (equation 9.14), it is given as:

$$ln\; p(X|\pi,\mu,\Sigma) = \sum_{n=1}^{N} ln \left\{ \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) \right\} \: \: (5)$$

where Bishop sums over the latent variable $$Z$$ and the observations $$x$$.Which looks an awful lot like my attempt at the log likelihood (3)

My questions:

1. Surely if the progression from (4) to (5) holds shouldn't the log likelihood be (3) rather than (1) ?

2. Did bishop just miss out the sum over the data?

(1) is the general formula obtained using marginalisation. In (2), you assume iid data samples, so the joint distribution transforms into multiplication of marginals, e.g. $$p(\mathbf{X})=p(x_1,x_2)=p(x_1)p(x_2)$$ In the book, the sentence before equation 9.14, equation (5) in your question, states that the dataset is assumed to be iid.
• 1. The joint $p(X,Z|\theta)$ is marginalised over $Z$ to obtain $p(X|\theta)$ 2. (1) doesn't assume iid samples, it's generic, i.e. $p(X)\neq \prod p(x)$ always. – gunes Mar 25 '20 at 23:19
• Right! I think I understand. Should I always assume, when I come across things in the future data is NOT IID unless it says so? (Particularly if I see $p(X)$ rather than $\prod p(x)$ (Maybe this is a silly question...) – RNs_Ghost Mar 25 '20 at 23:24