12
$\begingroup$

I am struggling to make the mathematical connection between a neural network and a graphical model.

In graphical models the idea is simple: the probability distribution factorizes according to the cliques in the graph, with the potentials usually being of the exponential family.

Is there an equivalent reasoning for a neural network? Can one express the probability distribution over the units (variables) in a Restricted Boltzmann machine or a CNN as a function of their energy, or the product of the energies between units?

Also, is the probability distribution modelled by an RBM or Deep belief network (e.g. with CNNs) of the exponential family?

I am hoping to find a text that formalizes the connection between these modern types of neural networks and statistics in the same way that Jordan & Wainwright did for graphical models with their Graphical Models, Exponential Families and Variational Inference. Any pointers would be great.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ IM (hater's) O the core problem here is that neural networks are not really networks; they practically have a fixed topology and thus have a minor chance to store any information inside it. $\endgroup$
    – user88
    Commented May 7, 2013 at 15:35
  • $\begingroup$ Have you seen this recent post? $\endgroup$
    – jerad
    Commented May 7, 2013 at 16:27
  • $\begingroup$ @jerad Thanks, I hadn't read that post. My question is not so much on how to combine these models (e.g. such as when Yann says "using deep nets as factors in an MRF"), but more about how to look at a deep net as a probabilistic factor graph. When Yann LeCun's says "of course deep Boltzmann Machines are a form of probabilistic factor graph themselves", I am interested in seeing that connection mathematically. $\endgroup$
    – Amelio Vazquez-Reina
    Commented May 7, 2013 at 16:56
  • $\begingroup$ @mbq, we've seen some forms of hidden layer component information storage, eg https://distill.pub/2017/feature-visualization/ (How neural networks build up their understanding of images), in that a complex image has component objects represented by hidden layer nodes. The weights can 'alter' the 'topology' in a non-discrete fashion. Although I have not seen it, some methods could include shrinkage factors to remove edges and therefore change the original topology $\endgroup$
    – Vass
    Commented Mar 23, 2019 at 17:48

3 Answers 3

Reset to default
6
$\begingroup$

Another good introduction on the subject is the CSC321 course at the University of Toronto, and the neuralnets-2012-001 course on Coursera, both taught by Geoffrey Hinton.

From the video on Belief Nets:

Graphical models

Early graphical models used experts to define the graph structure and the conditional probabilities. The graphs were sparsely connected, and the focus was on performing correct inference, and not on learning (the knowledge came from the experts).

Neural networks

For neural nets, learning was central. Hard-wiring the knowledge was not cool (OK, maybe a little bit). Learning came from learning the training data, not from experts. Neural networks did not aim for interpretability of sparse connectivity to make inference easy. Nevertheless, there are neural network versions of belief nets.

My understanding is that belief nets are usually too densely connected, and their cliques are too large, to be interpretable. Belief nets use the sigmoid function to integrate inputs, while continuous graphical models typically use the Gaussian function. The sigmoid makes the network easier to train, but it is more difficult to interpret in terms of the probability. I believe both are in the exponential family.

I am far from an expert on this, but the lecture notes and videos are a great resource.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Welcome to the site. We are trying to build a permanent repository of high-quality statistical information in the form of questions & answers. Thus, we're wary of link-only answers, due to linkrot. Can you post a full citation & a summary of the information at the link, in case it goes dead? $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 27, 2015 at 14:32
  • $\begingroup$ This is really nice. Thank you for adding this information & welcome to CV. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 27, 2015 at 16:43
  • $\begingroup$ I have to point out that the information in the first half of your answer is not quite accurate, which I guess is implied by the use of "early graphical models" (should be "very very early"). For a very long time, graphical models have been used to learn all aspects of its architecture the same way as neural networks have. But your later suggestion on sigmoids taking the place of gaussians in factor graphs is interesting! $\endgroup$
    – GuSuku
    Commented Nov 20, 2015 at 19:21
4
$\begingroup$

Radford Neal has done a good bit of work in this area that might interest you, including some direct work in equating Bayesian graphical models with neural networks. (His dissertation was apparently on this specific topic.)

I'm not familiar enough with this work to provide an intelligent summary, but I wanted to give you the pointer in case you find it helpful.

Improve this answer
$\endgroup$
2
  • $\begingroup$ From what I understand from works of Neal, Mackay etc, they are using Bayesian Optimization where the parameters to optimize over are the neural weights and biases, even going to show that the L2 normalization of neural networks can be seen as a Gaussian prior over the weights. That program has been continued to include number of hidden layers, neurons within each layer etc among the optimization variables. $\endgroup$
    – GuSuku
    Commented Nov 20, 2015 at 19:11
  • $\begingroup$ But this is different from what the OP asked because designing the architecture of the neural network to tryout in the next run is just one special case of experimental design using Bayesian models as a hyper-design engine. I think what the OP asked for was a mapping between neural network and bayesian modeling, at the "same level". $\endgroup$
    – GuSuku
    Commented Nov 20, 2015 at 19:14
4
$\begingroup$

This may be a old thread, but still a relevant question.

The most prominent example of the connections between Neural Networks (NN) and Probabilistic Graphical Models (PGM) is the one between Boltzmann Machines (and its variations like Restricted BM, Deep BM etc) and undirected PGMs of Markov Random Field.

Similarly, Belief Networks (and it's variations like Deep BN etc) are a type of directed PGMs of Bayesian graphs

For more, see:

  1. Yann Lecun, "A tutorial on energy-based learning" (2006)
  2. Yoshua Bengio, Ian Goodfellow and Aaron Courville, "Deep Learning", Ch 16 & 20 (book in preparation, at the time of writing this)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.