In the second paragraph of Section 11.3 in Machine Learning A Probabilistic Perspective, the author concisely summarizes Section 10.4.2 by saying that for the standard bayesian model $$P({\boldsymbol\theta}|D)=\frac{P(D|{\boldsymbol\theta})P({\boldsymbol\theta})}{P(D)}$$

...when we have complete data and a factored prior, the posterior over the parameters also factorizes...

Let's take a simple example. A single observation from a bivariate normal with known covariance matrix. The associated graphical model I have in mind is $$\mu_1\rightarrow x_1\rightarrow x_2\leftarrow \mu_2$$ where $\mu=(\mu_1,\mu_2)$ are the means and $x=(x_1,x_2)$ is the single observed data point. Note that this graphical model does in fact encode the desired independence: $(\mu_1\perp\mu_2)\;|\;(x_1,x_2)$.

For simplicities sake let's make the prior distribution uniform. Thus we have $$P(\mu)=N(0,\infty I)$$ $$P(x|\mu)=N(\mu, \Sigma)$$ where we choose the covariance matrix of the sample distribution to be $$\Sigma=\begin{bmatrix} 1 &\frac{1}{2} \\ \frac{1}{2} &1 \end{bmatrix}$$

Then the posterior distribution for $\mu$ is given by $$P(\mu|x)=N(x, \Sigma)$$ Here's the rub. The prior is clearly factorable, and the data is completely observed, yet the posterior is not factorable, since its covariance matrix is not diagonal.

So I must be incorrectly interpreting what the author means by the term 'complete data'. Could someone help me correct my misunderstanding here, it's been driving my nuts.

  • $\begingroup$ @Xi'an, do you mean my graphical model does not encode this independence, or that my example doesn't? $\endgroup$ – Thoth Jan 31 '16 at 8:41
  • $\begingroup$ @Xi'an as far as I can tell my book is telling me it is, as per figure 11.9 on page 345 (section 11.3). And it makes sense intuitively that it would be, since how can knowing $\mu_2$ affect my knowledge of $\mu_1$ when $x_1$ is fixed? $\endgroup$ – Thoth Jan 31 '16 at 8:53
  • $\begingroup$ is the quote I cited by the author also wrong? I'm sorry I'm just very confused at this point. $\endgroup$ – Thoth Jan 31 '16 at 8:56
  • $\begingroup$ @Xi'an just added parentheses around the independence of $\mu_1$ and $\mu_2$, I forgot to add that, that may make your interpretation of what I was trying to say different, my apologies. $\endgroup$ – Thoth Jan 31 '16 at 9:02

The notion of complete data is very weak, in that it simply means you observe the original model, rather than incomplete data. There is no conditional independence assumption behind this notion. In that sense, the quote from Section 11.3 is clearly confusing.

In equation (10.27) of Murphy'sMachine Learning, the sampling model is such that it factorises as $$\prod_t p(\mathcal{D}_t|\theta_t)$$ And Murphy rightly states that when the prior also factorises as $$\prod_t p(\theta_t)$$ the posterior reproduces this factorisation $$p(\theta|\mathcal{D})\propto \prod_t p(\mathcal{D}_t|\theta_t)p(\theta_t)$$ i.e. the $\theta_t$'s are independent a posteriori.

In your graphical model and Gaussian example, there is no such separation in the sampling model: $\mu_1$ and $\mu_2$ are independent a priori (if we ignore the issue with the infinite mass prior) but not a posteriori.

  • $\begingroup$ Yes I can see that that factorization leads to a factored posterior, but what about the graphical model also at the top of section 11.3, he states that if the $z_i$ was observed then by d-seperation $\theta_x\perp\theta_z|D$. $\endgroup$ – Thoth Jan 31 '16 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.