5
$\begingroup$

I have been looking at Bayesian networks, using the book by Koller, Probabilistic Graphical Models. The book is very large, so I might have missed the answer to my question ;). But I was trying to understand what the applications or outputs of a Bayesian network are.

I understand that we can factor the joint probability of some system of measurements and outcomes into a more limited set of conditional probabilities. And I understand how different flows of influence arise as we observe certain variables in the network versus other unknown quantities in the network.

What I am not clear about, is what the network is actually giving as output. It seems like the basic idea is to provide some classification of a particular state given the data. Take the simplest case where we see that the grass is wet and we want to know whether it rained or whether the sprinklers were used earlier. In this case, the network just indicates the probability of each possible state--given a list of states.

But beyond this, what else do Bayesian networks do? Do they provide predictions such as the value of some output variable given some data. I keep hearing about object tracking applications of Bayesian networks, but I wanted to understand what is actually provided. So there are noisy sensors that give an indication of some object's location. Now using a Bayesian network, does the network tell you the adjusted location of the object--similar to a Kalman filter might do. I mean does it tell you the adjusted coordinates, or does it simply tell you the probability of being in some discrete range of states?

I guess I am just trying to understand the forest for the tree, when it comes to Bayesian networks. Any help is appreciated.

$\endgroup$

1 Answer 1

4
$\begingroup$

You are correct in stating that a Bayesian network allows us to answer probability queries. However, the way you have framed the question suggests as if somehow you expected them to provide you with something more than that. And that it may not be clear to you how or why Bayesian networks are at all useful.

The main utility of Bayesian networks is that they provide a visual representation of what can be complex dependencies in a joint probability distribution - nodes represent random variables, and edges encode dependencies between random variables. Or more precisely, they encode conditional independences between random variables.

Representing complex dependencies.

At a high-level and without getting into details, this is a significant move in that it makes visualising complex joint distributions and the probabilistic relationships between the random variables at least possible - consider trying to represent a joint distribution over tens of thousands of random variables in bioinformatics applications using only algebra.

Bridging probabilistic inference with computer science.

However, there is a deeper consequence of the visual representation - it provides an interface for assessing whether probabilistic inference on a specified model is tractable. Probabilistic inference is the main activity associated with Bayesian networks, and by that, I mean computing the marginal or conditional probability of a single random variable $p(X_i)$, $p(X_i | X_{-i})$ or a set of random variables, from a joint distribution $p(X_1, ..., X_n)$ given an associated factorisation, and observed values of the other random variables $X_{-i}$

This is significant in that by turning the algebraic representation of a joint probability distribution $p(X_1, ..., X_n)$ into a data-structure on which we can operate, probabilistic inference (i.e. marginalisation) can itself then be viewed as a purely graph-theoretic, algorithmic procedure, e.g. message passing on the data-structure itself). Complex sequences of algebraic manipulations e.g. marginalising out variables using the sum-product operation, which would be infeasible to get one's head around without the graphical model interface, are now just operations on graphs. A further consequence is that we can quantify the time and space complexity of probabilistic inference, just as if it were any other computer science algorithm - we can evaluate tractability, and perhaps design more efficient algorithms etc etc.

In scientific fields where probabilistic and statistical arguments are marshalled, one of the constraints on our ability to empirically test a claim we have articulated is the availability of data. The additional constraint that the algorithmic view Bayesian networks (and graphical models in general) usefully provide(s) is that the computational feasibility of probabilistic inference over candidate dependency structures between many variables allows us to rule out dependency structures which we can articulate theoretically, but are simply not able to practicably test given a reasonable timeframe and finite resources for computation.

As a cognitive technology.

These last few points are speculative. I think there are good reasons to suspect that it is in the very representational power of graphical models as a visual interface that allowed for certain similarities across different domains of knowledge to be identified, thereby spurring the cross-transfer of techniques from other fields into machine learning.

As one example, note that in statistical physics, theoreticians were hindered by an incomputable partition function of the Boltzmann distribution over an exponential number of magnetic states in the Ising model of magnetism. Their response was to develop approximation techniques known as mean-field theory, which simplified the otherwise intractable combinations of spin interactions in the model. This allowed them to lower bound the partition function using variational principles, known as the Gibbs-Bogoliubov-Feynman inequality. This is exactly where variational inference in machine learning has come from - through graphical models.

To close, I would say that it's these kinds of characteristics of graphical models which would make them a good candidate for being considered, from a cybernetic perspective, as a cognitive technology in the sense of this article here, rather than just a visual interface.


I came across some information that I was previously unaware of when I wrote the above, so here is a quotation which I found by Michael Jordan, on Kevin Murphy's homepage, from 23 years ago:

"Graphical models are a marriage between probability theory and graph theory. They provide a natural tool for dealing with two problems that occur throughout applied mathematics and engineering -- uncertainty and complexity -- and in particular they are playing an increasingly important role in the design and analysis of machine learning algorithms. Fundamental to the idea of a graphical model is the notion of modularity -- a complex system is built by combining simpler parts. Probability theory provides the glue whereby the parts are combined, ensuring that the system as a whole is consistent, and providing ways to interface models to data. The graph theoretic side of graphical models provides both an intuitively appealing interface by which humans can model highly-interacting sets of variables as well as a data structure that lends itself naturally to the design of efficient general-purpose algorithms.

Many of the classical multivariate probabilistic systems studied in fields such as statistics, systems engineering, information theory, pattern recognition and statistical mechanics are special cases of the general graphical model formalism -- examples include mixture models, factor analysis, hidden Markov models, Kalman filters and Ising models. The graphical model framework provides a way to view all of these systems as instances of a common underlying formalism. This view has many advantages -- in particular, specialized techniques that have been developed in one field can be transferred between research communities and exploited more widely. Moreover, the graphical model formalism provides a natural framework for the design of new systems."

$\endgroup$
2
  • $\begingroup$ this is a perfect response, very thoughtful. I totally understand what you mean by answering probability queries and also providing a method to determine the order of obtaining an answer to that query--given the structure of conditional independence and dependence in the graph. This was very helpful. $\endgroup$
    – krishnab
    Mar 15, 2021 at 20:01
  • 1
    $\begingroup$ @krishnab. It turns out that Michael Jordan and his lab in the 90s were well aware of and contributed to the transfer of techniques normally used in statistical mechanics i.e. variational inference, into graphical models. I have updated the answer with a quote I found, in case it is of interest. $\endgroup$
    – microhaus
    Jun 8, 2021 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.