# With complete data and a factored prior, the posterior also factors

In the second paragraph of Section 11.3 in Machine Learning A Probabilistic Perspective, the author concisely summarizes Section 10.4.2 by saying that for the standard bayesian model $$P({\boldsymbol\theta}|D)=\frac{P(D|{\boldsymbol\theta})P({\boldsymbol\theta})}{P(D)}$$

...when we have complete data and a factored prior, the posterior over the parameters also factorizes...

Let's take a simple example. A single observation from a bivariate normal with known covariance matrix. The associated graphical model I have in mind is $$\mu_1\rightarrow x_1\rightarrow x_2\leftarrow \mu_2$$ where $\mu=(\mu_1,\mu_2)$ are the means and $x=(x_1,x_2)$ is the single observed data point. Note that this graphical model does in fact encode the desired independence: $(\mu_1\perp\mu_2)\;|\;(x_1,x_2)$.

For simplicities sake let's make the prior distribution uniform. Thus we have $$P(\mu)=N(0,\infty I)$$ $$P(x|\mu)=N(\mu, \Sigma)$$ where we choose the covariance matrix of the sample distribution to be $$\Sigma=\begin{bmatrix} 1 &\frac{1}{2} \\ \frac{1}{2} &1 \end{bmatrix}$$

Then the posterior distribution for $\mu$ is given by $$P(\mu|x)=N(x, \Sigma)$$ Here's the rub. The prior is clearly factorable, and the data is completely observed, yet the posterior is not factorable, since its covariance matrix is not diagonal.

So I must be incorrectly interpreting what the author means by the term 'complete data'. Could someone help me correct my misunderstanding here, it's been driving my nuts.

• @Xi'an, do you mean my graphical model does not encode this independence, or that my example doesn't?
– Set
Jan 31, 2016 at 8:41
• @Xi'an as far as I can tell my book is telling me it is, as per figure 11.9 on page 345 (section 11.3). And it makes sense intuitively that it would be, since how can knowing $\mu_2$ affect my knowledge of $\mu_1$ when $x_1$ is fixed?
– Set
Jan 31, 2016 at 8:53
• is the quote I cited by the author also wrong? I'm sorry I'm just very confused at this point.
– Set
Jan 31, 2016 at 8:56
• @Xi'an just added parentheses around the independence of $\mu_1$ and $\mu_2$, I forgot to add that, that may make your interpretation of what I was trying to say different, my apologies.
– Set
Jan 31, 2016 at 9:02

In equation (10.27) of Murphy'sMachine Learning, the sampling model is such that it factorises as $$\prod_t p(\mathcal{D}_t|\theta_t)$$ And Murphy rightly states that when the prior also factorises as $$\prod_t p(\theta_t)$$ the posterior reproduces this factorisation $$p(\theta|\mathcal{D})\propto \prod_t p(\mathcal{D}_t|\theta_t)p(\theta_t)$$ i.e. the $\theta_t$'s are independent a posteriori.
In your graphical model and Gaussian example, there is no such separation in the sampling model: $\mu_1$ and $\mu_2$ are independent a priori (if we ignore the issue with the infinite mass prior) but not a posteriori.
• Yes I can see that that factorization leads to a factored posterior, but what about the graphical model also at the top of section 11.3, he states that if the $z_i$ was observed then by d-seperation $\theta_x\perp\theta_z|D$.