Bayes network, what is the purpose of it?

Question

I really don't understand what is the purpose of the Bayes network, actually how to implement it into a useful application.

It all starts with the data. Let's assume I observe some universe and I have three random variables $A,B,C$. From there I can create a joint distribution table (see table below). As far as I understand it, this is my solid base, from there on I can use some probability laws and infer everything that I would like and I am done.

And now let's go to a problem. If I understand correctly Bayes networks are based on assumption. Therefore one can say (according to a data above), we can make the following model:

$P(A,B,C) = P(C|A,B) \cdot P(B|A) \cdot P(A)$

following conditional independence:

$P(A,B,C) = P(C|A,B) \cdot P(B) \cdot P(A)$

if we would infer $P(B|C,A) = P(C|A,B) \cdot P(B) / P(C|A)$

But just look the joint probability table above. Plug the data to formula and you would see how wrong this assumption is.

Just looking at the table: $P(B=1|C=1,A=1) = 0.31$

Using Bayes network:

$P(B=1| C= 1, A=1) = P(C=1|A=1,B=1) \cdot P(B=1) / P(C=1|A=1)$ results in 0.42

So my point is:

1. If you use joint probability table only, you get result from data, which is accurate and you don't have to assume anything. Computing gets more difficult if you have more random variables, but o.k...

2. To use Bayes network and create a graph, you need to make a very expensive assumption of your model. As more random variables you have, it's more likely you fail with assumption. I have presented you assumption on conditional independence can give you totally wrong result and it's a simple 3 random variable model. Now think if size would increase.

But, I know I fail to see something very important and I would appreciate if you could help me understand this better. Thank you.

Answer 1

Bayes Nets saves you from needing to estimate and store a large number of parameters. If you have $V$ variables with $D$ possible values each, to create the joint distribution table you would need to estimate and store $O(D^V)$ different parameters. In real problems with 100s of variables with many possible values each, this is clearly impractical. You'll never have enough data to create the table (and likely run out of computational resources as well).

By using the conditional Independence relations you can get away with estimating a much smaller number of parameters. But of course, as your example shows, you need to ensure that you have the conditional independence structure right - otherwise you will not get correct answers.

But at the same time in a real problem, you have limited amount of data, and your computing resources are not unbounded. So even if you get the structure wrong, Bayes net type approach may still be the only viable option. And of course, there are also methods to actually estimate the conditional independence structure - but learning the structure is generally considered difficult.

Edited to add: the wikipedia article seems quite good.