1
$\begingroup$

I am reading Chris Bishop's chapter on Expectation Propagation and there is the bit on how approximating factor graphs as shown in the images.

enter image description here

So, the original joint distribution can be written as:

$$ p(x) = f_a(x_1, x_2) \; f_b(x_2, x_3) \; f_c(x_2, x_4) $$

Now, the assumption is that each of these factors also factorise according to the variables they depend on, so we have an approximating distribution as:

$$ q(x) = \widetilde{f}_{a1}(x_1) \; \widetilde{f}_{a2}(x_2) \; \widetilde{f}_{b2}(x_2) \; \widetilde{f}_{c2}(x_2) \; \widetilde{f}_{b3}(x_3) \; \widetilde{f}_{c4}(x_4) $$

Now, just let us consider the original factor $f_a$. It has been split into $\widetilde{f}_{a1}(x_1)$ and $\widetilde{f}_{a2}(x_2)$. So, I am guessing the original factor which depended on both $x_1$ and $x_2$ can be approximated as a product of two unary variables i.e. $f_a(x_1, x_2) \approx \widetilde{f}_{a1}(x_1) \widetilde{f}_{a2}(x_2)$.

Now imagine that the original factor $f_a$ represented some joint probability as $P(x_1|x_2) P(x_2)$. Now imagine this $P(x_1|x_2)$ is a multivariate normal and $P(x_2)$ is a gamma distributed. They both represent priors in the joint distribution. Now, how does this relate to the approximating factors $\widetilde{f}_{a1}(x_1)$ and $\widetilde{f}_{a2}$? Can I give them whatever distributions I find suitable? For example, can I make $\widetilde{f}_{a1}(x_1)$ a multivariate normal and $\widetilde{f}_{a2}$ as a Gamma distribution? What about any other arbitrary distribution? Is it unto the user discretion to choose the appropriate factor forms?

The reason I ask this is that ultimately the full approximate joint distribution has a form typically a multidimensional normal distribution and there must be some constraints on what can be chosen for these factor messages? This is on page 514-515 of the Bishop book. he does not speak much about the form of these factor messages?

I did ask a similar question before but it did not get much attention. I tried to make some progress on it since then but so far have been unsuccessful.

$\endgroup$
1
+100
$\begingroup$

The right side figure corresponds to a fully factorized approximation. Within such approximations, the only constraint is that the messages into a variable must be from the same exponential family. The messages into a variable are the same as the factors attached to a variable in the right side figure. So if $\tilde{f}_{a2}$ is Gamma then $\tilde{f}_{b2}$ and $\tilde{f}_{c2}$ must also be Gamma. Alternatively, these three could all be Gaussian, or they could all be Beta. The family is for you to decide and is not dictated by the original factors. Also see my answer to Approximating distributions in expectation propagation.

Your question also mentions multidimensional normal distributions. However, those are not fully factorized. So the right side figure cannot be depicting a multidimensional normal distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.