# Derivation of EM in bishop

I'm working through chapter 9 in Bishop (Mixture models and EM) and I'm stuck on equation 9.29. For those without the book:

Bishop states that the log likelihood for a latent variable model with

1. data, $$X$$
2. latent variables, $$Z$$
3. latent model parameters $$\theta$$

is given by:

$$ln \; p(\textbf{X}| \boldsymbol{\theta}) = ln \; \left\{ \sum_{\textbf{Z}} p(\textbf{X,Z}|\boldsymbol{\theta} ) \right\} \: \: (1)$$

Shouldn't the log likelihood be over the data as well as the latent variables? Every example I've seen of a likelihood is over the data. If this is the case shouldn't the likelihood be:

$$p(\textbf{X}| \boldsymbol{\theta}) = \prod_{ \textbf{x} \in \textbf{X}} \sum_{\textbf{Z}} p(\textbf{x,Z}|\boldsymbol{\theta} ) \: \: (2)$$

(multiplying over the data)

$$ln \: p(\textbf{X}| \boldsymbol{\theta}) = \sum_{ \textbf{x} \in \textbf{X} } ln \left\{ \sum_{\textbf{Z}} p(\textbf{x,Z}|\boldsymbol{\theta} ) \right\} \: \: (3)$$

where $$\textbf{x}$$ is one observation in $$\textbf{X}$$

It strikes me that (1) is the equivalent of equation $$9.7$$ in Bishop (p430)

$$p(\textbf{x}) = \sum_{k=1}^K \pi_k {N}(\textbf{x}| \boldsymbol{\mu} , \boldsymbol{\sigma}) \: \: (4)$$

Which is not described as a likelihood. When (4) is turned into a likelihood later on (equation 9.14), it is given as:

$$ln\; p(X|\pi,\mu,\Sigma) = \sum_{n=1}^{N} ln \left\{ \sum_{k=1}^K \pi_k N(x_n|\mu_k,\Sigma_k) \right\} \: \: (5)$$

where Bishop sums over the latent variable $$Z$$ and the observations $$x$$.Which looks an awful lot like my attempt at the log likelihood (3)

My questions:

1. Surely if the progression from (4) to (5) holds shouldn't the log likelihood be (3) rather than (1) ?

2. Did bishop just miss out the sum over the data?

## 1 Answer

(1) is the general formula obtained using marginalisation. In (2), you assume iid data samples, so the joint distribution transforms into multiplication of marginals, e.g. $$p(\mathbf{X})=p(x_1,x_2)=p(x_1)p(x_2)$$ In the book, the sentence before equation 9.14, equation (5) in your question, states that the dataset is assumed to be iid.

• Thanks again @gunes. Two questions: 1. What has been marginalised over in equation (1)? 2. Does equation (1) NOT assume IID samples then? – RNs_Ghost Mar 25 '20 at 23:17
• 1. The joint $p(X,Z|\theta)$ is marginalised over $Z$ to obtain $p(X|\theta)$ 2. (1) doesn't assume iid samples, it's generic, i.e. $p(X)\neq \prod p(x)$ always. – gunes Mar 25 '20 at 23:19
• Right! I think I understand. Should I always assume, when I come across things in the future data is NOT IID unless it says so? (Particularly if I see $p(X)$ rather than $\prod p(x)$ (Maybe this is a silly question...) – RNs_Ghost Mar 25 '20 at 23:24
• Yes, iid assumption is typical but extra. – gunes Mar 25 '20 at 23:56