Specifying conditional probabilities in hybrid Bayesian networks

Question

I am trying to get a deeper understanding of the various types of Bayesian networks.

Most of the literature/lectures I've come across use discrete random variables exclusively and only mention continuous random variables in passing.

It seems if you want to mix discrete and continuous variables in a hybrid network, then you have a few different scenarios to handle:

Child Is       Parent(s) Are
----------------------------------------
Discrete       All Discrete
Discrete       All Continuous
Discrete       Hybrid (Discrete and Continuous)
Continuous     All Discrete
Continuous     All Continuous
Continuous     Hybrid (Discrete and Continuous)

I understand how to specify conditional probabilities (i.e. $p(N=n_i|P)$) for the cases where the parents (P) are all discrete. The conditional probabilities are essentially a lookup table that maps a condition to either a probability mass function (when child is discrete) or probability density function (when child is continuous).

The other cases are numbing my brain. How does one go about specifying conditional probabilities of a given node when any of its parents are continuous?

Thanks

Edit Terminology has been (hopefully) corrected based on feedback.

Answer 1

First of all, your usage of the term "prior probability" seems to be wrong. For any node N with discrete values $n_i$ the probability that a certain value of N occurs a priori is $p(N=n_i)$. If a node has no parents, one is interested in calculate this prob. But if a node has parents P, one is interested in calculating the conditional probability, i.e. the probability that a certain value of N occurs given its parents. This is $p(N=n_i|P)$.

Regarding your actual question: How to calculate the conditional probabilities of a child node if the parents are continuous and the child node is a) discrete or b) continuous. I will to explain one approach using a concrete example, but I hope the general idea will be clear though.

Consider this network:

where subsidy and buys are discrete nodes meanwhile harvest and cost are continuous.

a) p(Cost|Subsidy, Harvest) Two options:

1. Discretize Harvest and treat it as discrete => maybe information loss
2. Model a mapping from the current value of harvest to a parameter describing the distribution of Cost.

Details to 2.: Let's assume cost can be modeled using a normal distribution. In this case it is common to fix the variance and map the value of Harvest linearly to the mean of gaussian. The parent subsidy (binary) only add the constraint to create a separate distribution for subsidy=true and subsidy=false.

Result: $p(Cost=c|Harvest=h,Subsidy=true)=N(\alpha_1 * h + \beta_1,\sigma_1^2)(c)$

$p(Cost=c|Harvest=h,Subsidy=false)=N(\alpha_2 * h + \beta_2,\sigma_2^2)(c)$

for some $\alpha$s and $\beta$s.

b) p(Buys|Cost) In this case on need to map the probability of occurrence of certain costs to the probability of Buy=True (p(Buy=False) = 1-p(Buy=True)). (Note that this is the same task as in logistic regression). One approach is if the parent has a normal distribution, to calculate the integral from 0 to x of a standard normal distribution, where x is the z-transformed value of the parent. In our example: $p(Buys=true|Cost=c) = integral(0,\frac{-c+\mu}{\sigma})$ with $\mu$=mean and $\sigma$=standard deviation of the Cost distribution. In this case $-c+\mu$ instead of $\mu-c$, because an observation (knowledge extracted from data !) is, that the lower the cost the more probable is a buy.

In the case of a non-binary discrete child-node, one approach is to transform the one multi-value-problem into multiple binary problems.

Example-Source / Further reading:

"Artificial Intelligence: A modern approach" by Russell and Norvig

Answer 2

Consider two simple cases,

1) a real valued variable X is the parent of another real valued variable Y
2) a real valued variable X is the parent of a discrete valued variable Y

Assume that the Bayes net is a directed graph X -> Y. The Bayes net is fully specified, in both cases, when P(X) and P(Y | X) are specified. Or, strictly speaking when P(X | a) and P(Y | X, b) are specified where a and b are the parameters governing the marginal distribution of X and the conditional distribution of Y given X, respectively.

Parametric Assumptions

If we are happy for some reason to assume that P(X | a) is Normal, so a contains its mean and its variance.

Consider case 1. Perhaps we are happy to assume that P(Y | X, b) is also Normal and that the conditional relationship is linear in X. Then b contains regression coefficients, as we normally understand them. That means here: an intercept, a slope parameter, and a conditional variance.

Now consider case 2. Perhaps we are willing to assume that P(Y | X, b) is Binomial and that the conditional relationship is linear in X. Then b contains the coefficients from a logistic regression model: here that means an intercept and a slope parameter.

In case 1 if we observe Y and try to infer X then we simply use Bayes theorem with the parametric assumptions noted above. The result is rather well known: the (posterior) mean of X given Y is a weighted average of the prior mean of X (gotten from a) and the observed value of Y, where the weights are a function of the standard deviations of X (from a) and Y given X (from b). More certainty about X pulls the posterior mean towards its mean.

In case 2 there may be no handy closed form for P(X | Y) so we might proceed by drawing a sample from P(X), making use of your assumptions about P(Y | X) and normalising the result. That is then a sample from P(X | Y).

In both cases, all we are doing is computing Bayes theorem, for which we need some parametric assumptions about the relevant distributions.

All this gets more complicated when you want to learn a and b, rather than assuming them, but that is, I think, beyond your question.

Sampling

When thinking about continuous parents it may be helpful to think about generating data from the network. To make on data point form the network in case 1 (with the assumptions as above) we would first sample from P(X) - a Normal distribution - then take that value and generate a conditional distribution over Y by plugging the value we generated into the regression model of P(Y | X) as an independent variable. That conditional distribution is also Normal (by assumption) so we then sample a Y value from that distribution. We now have a new data point. The same process applies when P(Y | X) is a logistic regression model. The model gives us a probability of 1 vs. 0 given the value of X we sampled. We then sample from that distribution - essentially tossing a coin biased according to the model probabilities - and we have a new data point. When we think about inference in a Bayes net, we are, conceptually speaking, just reversing this process.

Perhaps that helps?