I've been trying to get my hands on a substantial resource for using Gibbs sampling in hybrid Bayesian networks, that is, networks with both continuous and discrete variables. So far I can't say I have succeeded. I'm interested in hybrid networks where there are no constraints regarding discrete children having continuous parents. Gibbs sampling is a very widely employed method in approximate inference in Bayesian methods, and yet, I can't seem to find detailed resources that focus on hybrid networks

up vote 5 down vote accepted

Short answer: Gibbs or Metropolis-Hastings-within-Gibbs (MCMC) should work just fine on joint distributions and full conditional distributions that are mixed products of pmfs and pdfs. If you're doing MCMC, just make sure that sampling from the candidate distributions gives you values in the right domain.

Long answer:

The most comprehensive way we know of to account for probability is to use measure theory. In measure-theoretic probability theory, there is little difference between pmfs and pdfs.

In measure-theoretic probability, pmfs and pdfs are both called "densities." More precisely, they are both Radon-Nikodym (pronounced "RahDOHN-NickohDEEM") derivatives. The only difference is that a pmf is a derivative with respect to a counting measure, and a pdf is a derivative with respect to Lebesgue ("LehBAYG") measure (i.e. n-dimensional volume). When you integrate under a pmf, you integrate with respect to counting measure: you sum over part of its domain. When you integrate under a pdf, you integrate with respect to Lebesgue measure. In every case that Bayesians tend to care about, the latter is equivalent to regular old Riemann integration.

Because pmfs and pdfs are the same kind of thing, they obey the same laws, such as Bayes' law and the product rule. Thus, you can use both pmfs and pdfs in a model and get a meaningful joint density and meaningful conditional densities. But these densities won't necessarily be pdfs or pmfs. In general, they will be Radon-Nikodym derivatives with respect to products of counting and Lebesgue measures.

Riemann integration can't handle Radon-Nikodym derivatives with respect to mixed spaces like that, but Lebesgue integration can. Sampling methods approximately carry out Lebesgue integration, so they just work.

You might be wondering, then, what doesn't just work. Well, the only thing I've mentioned so far are products of densities and integrating under those products. A random variable with a distribution that is the sum of a distribution with a pmf and a distribution with a pdf can cause problems.

Any easy way to get a sum like that is to use an "if." For example, say you have this model:

$X \sim \mathrm{Normal}(0,1)$

$Y \sim \mathrm{Geometric}(1/2)$

$B \sim \mathrm{Bernoulli}(1/2)$

and you define a new random variable $Z$ by $Z = X$ if $B = 0$, otherwise $Z = Y$. The distribution of $Z$ can't be represented by a pmf or a pdf: its cdf has a discontinuity at each positive integer.

You can still approximate $Z$'s distribution using samples. You can even use $Z$ as a parameter for another random variable's distribution, and answer conditional queries with a Gibbs sampler. But conditioning on $Z$ in a query will not work without measure-theoretic tools. Also, you might find your current tools, like the density version of Bayes' law, inadequate. (I haven't looked into it enough to say when.)

Another easy way to get a non-pmf/non-pdf distribution is to truncate a random variable; something like $W = X$ if $X \le 0$, otherwise $W = 0$. (Note that truncating $X$'s distribution won't give you $W$'s, because $P[W=0] = 1/2$. Truncated distributions and truncated random variables are very different things.) $W$ would give you the same problems as $Z$.

Other things off the cuff:

You might have a hard time getting good n-dimensional KDEs that have discrete axes, if you want them.

If you're doing Metropolis-Hastings, you might have a hard time coming up with candidate pmfs for categorical random variables.

--

If you want another way to think about it, consider what would happen if you transformed your model. Convert your pmfs into pdfs and floor the corresponding discrete random variables before using them as parameters in other random variables' distributions. The joint distribution that includes the un-floored random variables (but not the floored ones) obviously has a pdf.

The easiest way to get an equivalent model is to define each new pdf $d$ so that, if its corresponding pmf is $m$, $d(x) = m(\lfloor x \rfloor)$. Now suppose you created an MCMC sampler from the transformed model. It shouldn't take too long to convince yourself that the fractional part of a candidate sample for any originally discrete random variable contributes nothing to the accept/reject decision.

  • Thanks for the comprehensive answer. I can go out and hunt for resources for details of the second part, where you outline the problematic cases, but if you could provide me a couple of pointers, that'd be fantastic. Books, or even keywords for resources to get a better handling of the problems would be fantastic. Unfortunately, you don't get to see these in most books about Bayesian statistics, networks etc. – mahonya Mar 26 '11 at 10:15

I think this is still an open research question and there has been little consensus on the best way to do this.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.