4
$\begingroup$

I am trying to get a deeper understanding of the various types of Bayesian networks.

Most of the literature/lectures I've come across use discrete random variables exclusively and only mention continuous random variables in passing.

It seems if you want to mix discrete and continuous variables in a hybrid network, then you have a few different scenarios to handle:

Child Is       Parent(s) Are
----------------------------------------
Discrete       All Discrete
Discrete       All Continuous
Discrete       Hybrid (Discrete and Continuous)
Continuous     All Discrete
Continuous     All Continuous
Continuous     Hybrid (Discrete and Continuous)

I understand how to specify conditional probabilities (i.e. $p(N=n_i|P)$) for the cases where the parents (P) are all discrete. The conditional probabilities are essentially a lookup table that maps a condition to either a probability mass function (when child is discrete) or probability density function (when child is continuous).

The other cases are numbing my brain. How does one go about specifying conditional probabilities of a given node when any of its parents are continuous?

Thanks

Edit Terminology has been (hopefully) corrected based on feedback.

$\endgroup$
  • 1
    $\begingroup$ I may be misunderstanding this question, but it seems that you're asking how to specify a prior on a node value when the node has parents. But you don't need to. Only nodes without parents need priors. You may want priors on some parameters that define the mapping from parent(s) to child. But these are priors on parameters not priors on node values not on values. $\endgroup$ – conjugateprior Dec 19 '10 at 13:45
  • $\begingroup$ I think I'm missing a subtlety in the last part of your comment, but also think you may be close to making sense out of my question and resolving my confusion. With a fully discrete network, my understanding is that the prior probability of a child node is specified in terms of its parents' combined states. How this notion of dependence carries forward to a network with continuous variables is what confuses me. Does that give you enough to elaborate? Is there a simple two node (parent -> child) network that exemplifies this kind of 'mapping'? $\endgroup$ – Joe Holloway Dec 20 '10 at 23:06
  • $\begingroup$ Pick some function f(p1,p2,c), and define conditional probability as f(p1,p2,c)/sum_c f(p1,p2,c). Replace sum with integral when c is continuous $\endgroup$ – Yaroslav Bulatov Dec 23 '10 at 0:12
5
$\begingroup$

First of all, your usage of the term "prior probability" seems to be wrong. For any node N with discrete values $n_i$ the probability that a certain value of N occurs a priori is $p(N=n_i)$. If a node has no parents, one is interested in calculate this prob. But if a node has parents P, one is interested in calculating the conditional probability, i.e. the probability that a certain value of N occurs given its parents. This is $p(N=n_i|P)$.

Regarding your actual question: How to calculate the conditional probabilities of a child node if the parents are continuous and the child node is a) discrete or b) continuous. I will to explain one approach using a concrete example, but I hope the general idea will be clear though.

Consider this network:

alt text

where subsidy and buys are discrete nodes meanwhile harvest and cost are continuous.

a) p(Cost|Subsidy, Harvest) Two options:

  1. Discretize Harvest and treat it as discrete => maybe information loss
  2. Model a mapping from the current value of harvest to a parameter describing the distribution of Cost.

Details to 2.: Let's assume cost can be modeled using a normal distribution. In this case it is common to fix the variance and map the value of Harvest linearly to the mean of gaussian. The parent subsidy (binary) only add the constraint to create a separate distribution for subsidy=true and subsidy=false.

Result: $p(Cost=c|Harvest=h,Subsidy=true)=N(\alpha_1 * h + \beta_1,\sigma_1^2)(c)$

$p(Cost=c|Harvest=h,Subsidy=false)=N(\alpha_2 * h + \beta_2,\sigma_2^2)(c)$

for some $\alpha$s and $\beta$s.

b) p(Buys|Cost) In this case on need to map the probability of occurrence of certain costs to the probability of Buy=True (p(Buy=False) = 1-p(Buy=True)). (Note that this is the same task as in logistic regression). One approach is if the parent has a normal distribution, to calculate the integral from 0 to x of a standard normal distribution, where x is the z-transformed value of the parent. In our example: $p(Buys=true|Cost=c) = integral(0,\frac{-c+\mu}{\sigma})$ with $\mu$=mean and $\sigma$=standard deviation of the Cost distribution. In this case $-c+\mu$ instead of $\mu-c$, because an observation (knowledge extracted from data !) is, that the lower the cost the more probable is a buy.

In the case of a non-binary discrete child-node, one approach is to transform the one multi-value-problem into multiple binary problems.

Example-Source / Further reading:

"Artificial Intelligence: A modern approach" by Russell and Norvig

$\endgroup$
  • $\begingroup$ OK, you're right in my misuse of terminology. I've been incorrectly calling them priors or sometimes 'conditional priors' because I specify the conditional probability table as part of modeling the network. I'll update my question so hopefully future travelers aren't as confused. $\endgroup$ – Joe Holloway Dec 22 '10 at 17:06
  • $\begingroup$ That's nice. Does the other part of the answer helps you, too ? Or does it need some refinement ? $\endgroup$ – steffen Dec 22 '10 at 17:31
  • $\begingroup$ Yes, I think you've nailed it, but I haven't had time to fully digest it yet. I'll come back and mark accepted when I've had time to digest. $\endgroup$ – Joe Holloway Dec 22 '10 at 18:22
  • $\begingroup$ @steffen I have some questions about this. I created a new question for that. If you could take a look at it, I would be grateful. $\endgroup$ – Parag S. Chandakkar Jan 28 '15 at 7:04
  • $\begingroup$ Do we have to solve an over-determined set of equations to get $\alpha$ and $\beta$? $\endgroup$ – Parag S. Chandakkar Jan 28 '15 at 10:11
2
$\begingroup$

Consider two simple cases,

1) a real valued variable X is the parent of another real valued variable Y
2) a real valued variable X is the parent of a discrete valued variable Y

Assume that the Bayes net is a directed graph X -> Y. The Bayes net is fully specified, in both cases, when P(X) and P(Y | X) are specified. Or, strictly speaking when P(X | a) and P(Y | X, b) are specified where a and b are the parameters governing the marginal distribution of X and the conditional distribution of Y given X, respectively.

Parametric Assumptions

If we are happy for some reason to assume that P(X | a) is Normal, so a contains its mean and its variance.

Consider case 1. Perhaps we are happy to assume that P(Y | X, b) is also Normal and that the conditional relationship is linear in X. Then b contains regression coefficients, as we normally understand them. That means here: an intercept, a slope parameter, and a conditional variance.

Now consider case 2. Perhaps we are willing to assume that P(Y | X, b) is Binomial and that the conditional relationship is linear in X. Then b contains the coefficients from a logistic regression model: here that means an intercept and a slope parameter.

In case 1 if we observe Y and try to infer X then we simply use Bayes theorem with the parametric assumptions noted above. The result is rather well known: the (posterior) mean of X given Y is a weighted average of the prior mean of X (gotten from a) and the observed value of Y, where the weights are a function of the standard deviations of X (from a) and Y given X (from b). More certainty about X pulls the posterior mean towards its mean.

In case 2 there may be no handy closed form for P(X | Y) so we might proceed by drawing a sample from P(X), making use of your assumptions about P(Y | X) and normalising the result. That is then a sample from P(X | Y).

In both cases, all we are doing is computing Bayes theorem, for which we need some parametric assumptions about the relevant distributions.

All this gets more complicated when you want to learn a and b, rather than assuming them, but that is, I think, beyond your question.

Sampling

When thinking about continuous parents it may be helpful to think about generating data from the network. To make on data point form the network in case 1 (with the assumptions as above) we would first sample from P(X) - a Normal distribution - then take that value and generate a conditional distribution over Y by plugging the value we generated into the regression model of P(Y | X) as an independent variable. That conditional distribution is also Normal (by assumption) so we then sample a Y value from that distribution. We now have a new data point. The same process applies when P(Y | X) is a logistic regression model. The model gives us a probability of 1 vs. 0 given the value of X we sampled. We then sample from that distribution - essentially tossing a coin biased according to the model probabilities - and we have a new data point. When we think about inference in a Bayes net, we are, conceptually speaking, just reversing this process.

Perhaps that helps?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.