Are graphical models and Boltzmann machines related mathematically?

Question

While I have actually done some programming with Boltzmann machines in a physics class, I am not familiar with their theoretical characterization. In contrast, I know a modest amount about the theory of graphical models (about the first few chapters of Lauritzen's book Graphical Models).

Question: Is there any meaningful relationship between graphical models and the Boltzmann machine? Is the Boltzmann machine a type of graphical model?

Obviously the Boltzmann machine is a type of neural network. I have heard that some neural networks are mathematically related to graphical models and that some are not.

Related Questions on CrossValidated which don't answer my question:
This is similar to a previous question which has been asked before: What's the relation between hierarchical models, neural networks, graphical models, bayesian networks? but is more specific.

Moreover, the accepted answer to that question does not clarify my confusion -- even if the nodes in the standard graphical representation of a neural network do not represent random variables, that does not necessarily mean that no such representation exists. Specifically, I am thinking about how the nodes in the typical graphical representation of Markov chains represent the set of possible states rather than the random variables $X_i$, but one could also create a graph showing the conditional dependence relationships between the $X_i$, which shows that every Markov chain is in fact a Markov random field. The answer also says that neural networks (presumably including Boltzmann machines) are "discriminative", but doesn't go into more detail to explain what that claim means, nor is the obvious follow-up question "are graphical models not discriminative?" addressed. Likewise, the accepted answer links to Kevin Murphy's website (I actually read some of his PhD thesis when learning about Bayesian networks), but this website discusses only Bayesian networks and does not mention neural networks at all -- thus it fails to illuminate how they are different.

This other question is probably most similar to mine: Mathematically modeling neural networks as graphical models However, none of the answers were accepted, and likewise only give references but do not explain the references (e.g. this answer). While one day I will hopefully be able to understand the references, right now I am at a basic level of knowledge and would most appreciate an answer which is as simplified as possible. Also, the Toronto course linked to in the top answer (http://www.cs.toronto.edu/~tijmen/csc321/lecture_notes.shtml) addresses this, but not in very much detail. Furthermore, the notes for the one lecture which might answer my question are not available to the public.

March 25 Lecture 13b: Belief Nets 7:43. For this slide, keep in mind Boltzmann Machines. There, too, we have hidden units and visible units, and it's all probabilistic. BMs and SBNs have more in common than they have differences. 9:16. Nowadays, "Graphical Models" are sometimes considered as a special category of neural networks, but in the history that's described here, they were considered to be very different types of systems.

Answer 1

Boltzmann machines vs. restricted Boltzmann machines

AFAIK the Boltzmann machines is a type of graphical model, and the model that's related to neural networks is the restricted Boltzmann machines (RBM).

The difference between Boltzmann machines and restricted Boltzmann machines, from the book Machine Learning A Probabilistic Perspective

RBMs vs. neural netowrks

For RBMs (ref: A Practical Guide to Training Restricted Boltzmann Machines by Geoffrey Hinton) $$p(\mathbf{v},\mathbf{h})=\frac{1}{Z}\exp(\sum a_iv_i+\sum b_jh_j + \sum v_ih_jw_{ij})$$ $$p(h_j=1|\mathbf{v})=\sigma(b_j+\sum v_iw_{ij})$$ $$p(v_i=1|\mathbf{h})=\sigma(a_i+\sum h_jw_{ij})$$ where $\mathbf{v}$ and $\mathbf{h}$ correspond to the visible and hidden units in the above figure, and $\sigma()$ is the Sigmoid function.

The conditional probabilities are computed in the same form of network layers, so the trained weights of RBMs can be used directly as the weights of neural networks or as a starting point of training.

I think the RBM itself is more of a graphical model than a type of neural network, since it is undirected, it has well defined conditional independencies, and it uses its own training algorithms (e.g. contrastive divergence).

Answer 2

This just confirms/verifies the accepted answer, that Boltzmann machines are indeed a special case of graphical model. Specifically, this question is addressed on pp. 127-127 of Koller, Friedman, Probabilistic Graphical Models: Principles and Techniques, in Box 4.C.

One of the earliest types of Markov network models is the Ising model which first arose in statistical physics as a model for the energy of a physical system involving a system of interacting atoms... Related to the Ising model is the Boltzmann machine distribution... the resulting energy can be reformulated in terms of an Ising model (Exercise 4.12).

How the Ising model, originally a concept from the statistical mechanics literature, can be formulated as a graphical model is given in much detail in Example 3.1., Section 3.3., on pp. 41-43 of Wainwright, Jordan, Graphical Models, Exponential Families, and Variational Inference.

Apparently the Ising model was instrumental in the foundation of the field of graphical models during the late 1970's and early 1980's, at least based on what Steffen Lauritzen says in both the preface and introduction to his book, Graphical Models. This interpretation also seems supported by Section 4.8 in Koller and Friedman above-cited book.

The development of Boltzmann machines from the Ising model may have been an independent occurrence, based on that same section of Koller and Friedman as well, which claims that "Boltzmann machines were first proposed by Hinton and Sejnowski (1983)", which seems to have occurred after the initial work in developing Markov random fields as generalizations of the Ising model, although the work behind that paper could have begun much earlier than 1983.

---

My confusion regarding this relationship, when I wrote this question more than a year ago, stemmed from the fact that I first encountered both the Ising model, and the Boltzmann machine model for neurons, in the physics literature. As Koller and Friedman mention, the literature within the statistical physics community about the Ising model and related notions is truly vast.

In my experience it is also fairly insular, in the sense that while statisticians and computer scientists studying graphical models will mention how the field is related to statistical mechanics, no reference I have ever found from the statistical physics literature mentions the connections to other fields or tries to exploit it. (Hence causing me to doubt and be confused by the notion that there could be any such connections to other fields.)

For an example of the physicist's perspective on both the Ising model and the Boltzmann machine, see the textbook from the course where I first learned of it. It also mentions mean field methods, if I remember correctly, something discussed as well in the Jordan and Wainwright article cited above.