I am currently studying Bayesian Reasoning and Machine Learning by David Barber, the 4th chapter exercise 4.1 (p 79). The exercise is the following:

Exercise 4.1

  1. Consider the pairwise Markov network, $$p(x) = \phi(x_1,x_2)\phi(x_2,x_3)\phi(x_3,x_4)\phi(x_4,x_1)$$ Express in terms of $\phi$ the following: $$p(x_1|x_2,x_4), p(x_2|x_1,x_3),p(x_3|x_2,x_4),p(x_4|x_1,x_3)$$
  2. For a set of local distributions defined as $$p(x_1|x_2,x_4), p(x_2|x_1,x_3),p(x_3|x_2,x_4),p(x_4|x_1,x_3)$$ is it always possible to find a joint distribution $p(x_1,x_2,x_3,x_4)$ consistent with these local conditional distributions?

Now, I've done the whole first part. I have a solution manual and checked my derivations are correct. Here's the first expression:

$$p(x_1|x_2,x_4) = \frac{\sum_3\phi(x_1,x_2)\phi(x_2,x_3)\phi(x_3,x_4)\phi(x_4,x_1)}{\sum_{1,3}\phi(x_1,x_2)\phi(x_2,x_3)\phi(x_3,x_4)\phi(x_4,x_1)} \\ = \frac{\phi(x_1,x_2)\phi(x_4,x_1)\sum_3\phi(x_2,x_3)\phi(x_3,x_4)}{\sum_1\phi(x_1,x_2)\phi(x_4,x_1)\sum_3\phi(x_2,x_3)\phi(x_3,x_4)}\\ = \frac{\phi(x_1,x_2)\phi(x_4,x_1)}{\sum_1\phi(x_1,x_2)\phi(x_4,x_1)}$$

What I don't understand is the manual's answer to the second question, in particular how their equation is correct. I also don't understand the question itself.

Here's their answer:

It is not always possible to do this. One way to see this is to consider for example $$p_1(x_1|x_2,x_4)\sum_{x_1,x_3}p(x_1,x_2,x_3,x_4) = p(x_1,x_2,x_3,x_4)$$

What does $p_1$ mean?

How is this correct? Since as I've shown before

$$p(x_1|x_2,x_4) = \frac{\sum_3\phi(x_1,x_2)\phi(x_2,x_3)\phi(x_3,x_4)\phi(x_4,x_1)}{\sum_{1,3}\phi(x_1,x_2)\phi(x_2,x_3)\phi(x_3,x_4)\phi(x_4,x_1)} \\ \Leftrightarrow \\ p(x_1|x_2,x_4) = \frac{\sum_3 p(x_1,x_2,x_3,x_4)}{\sum_{1,3} p(x_1,x_2,x_3,x_4)} \\ \Leftrightarrow \\ p(x_1|x_2,x_4) \sum_{1,3} p(x_1,x_2,x_3,x_4)= \sum_3 p(x_1,x_2,x_3,x_4)$$

So how is that correct? Also, what does it mean for a distribution to be consistent exactly?


1 Answer 1


¡The reported exercise uses confusing notations in that $p(\cdot)$ denotes many different functions!

If one defines the conditional distributions$$p(x_1|x_2,x_4), p(x_2|x_1,x_3),p(x_3|x_2,x_4),p(x_4|x_1,x_3)$$arbitrarily, there is no reason these four (marginalised) conditional densities are compatible with a joint distribution. That is, there may exist no joint distribution such that these are its four conditionals.

On the other hand, the equation $$p(x_1|x_2,x_4)\sum_{x_1,x_3}p(x_1,x_2,x_3,x_4) = p(x_1,x_2,x_3,x_4)$$ is not correct, as the lhs integrates $x_3$ out, while the rhs exhibits an $x_3$. The correct equation is $$p(x_1|x_2,x_4)\underbrace{\sum_{x_1,x_3}p(x_1,x_2,x_3,x_4)}_\text{marginal of $(x_2,x_4)$} = \underbrace{p(x_1,x_2,x_4)}_\text{marginal of $(x_1,x_2,x_4)$}$$ assuming $X_1$ only depends on $(X_2,X_4)$, ie is conditionally independent of $X_3$.

According to the Hammersley-Clifford theorem, the joint (when it exists) is given by $$\dfrac{p(x_1,x_2,x_3,x_4)}{p(x^0_1,x^0_2,x^0_3,x^0_4)}= \dfrac{p(x_1|x_2,x_4)p(x_2|x_1^0,x_3)p(x_3|x_2^0,x_4)p(x_4|x_1^0,x_3^0)}{p(x_1^0|x_2,x_4)p(x_2^0|x_1^0,x_3)p(x_3^0|x_2^0,x_4)p(x^0_4|x^0_1,x^0_3)}$$ where $(x_1^0,x^0_2,x_3^0,x^0_4)$ is an arbitrary value (with positive probability). For the conditionals to be compatible, the rhs should factor as a function of $(x_1^0,x^0_2,x_3^0,x^0_4)$ times a function of $(x_1,x_2,x_3,x_4)$.

  • $\begingroup$ Hi, thank you for the answer! What does it mean for a density to be compatible with a joint distribution? $\endgroup$
    – user
    Jul 10, 2022 at 15:10
  • 1
    $\begingroup$ See my editing. For instance, $p(x_1|x_2)$ and $p(x_2|x_1)$ are compatible with a joint distribution if and only if$$\dfrac{p(x_1|x_2)}{p(x_2|x_1)}=a(x_1)b(x_2)$$ie the ratio is factorising as the product of a function of $x_1$ and a function of $x_2$. $\endgroup$
    – Xi'an
    Jul 10, 2022 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.