I'm trying to implement an approximate inference algorithm based on junction tree algorithm for a Bayesian Network that has continuous variables which happen to have non-linear relationships, and in general their Conditional Probability Distributions (CPDs) are non-Gaussian and multi-modal.
To estimate and represent the CPD for a variable Y with parents X, I first estimate the joint distribution P(Y, X) from data as a Gaussian Mixture Model (GMM). Then using the GMM, I analytically compute the CPD P(Y | X) as a Mixture of Linear Gaussians where each component is a linear gaussian and the mixing weight of each component is also a function of X. (This seems similar to mixture of experts except that the mixing weights have gaussian kernels.) Ref: Chap. 4 in HG Sung's thesis.
Regarding inference, the junction tree (aka clique tree) algorithm involves product and sum operations on the messages/CPDs. While marginalization (sum) is feasible on the mixture, I can't seem to find an effective way to approximate the product of the mixtures of linear gaussians. Note that computing the exact product is usually not computationally tractable due to large number of product components - O($K^N$) for N input mixtures each with K components.
The Nonparametric Belief Propagation paper approximates the product of GMMs using Gibbs Sampling but I haven't been able to figure that out for mixture of linear gaussians.
Another approach is to reduce the number of components in the product mixture by pruning or merging components using some criteria. For a GMM, these algorithms typically look at the mixing weights and covariances, and decide which components to merge/prune. Since the mixing weights in the mixture of linear gaussians are a function (of X) rather than a value, I can't get this approach to work either.
My question is two-fold:
- What's a good way to approximate the product of mixtures of linear gaussians?
- In general, what are the preferred methods/algorithms to represent the CPDs, and perform (approximate) inference on a Bayesian Network with continuous variables with non-Gaussian CPDs?