Meaning of Probability Distributions in RBMs

Question

I'm new to machine learning, and am trying to understand some of the basics of Restricted Boltzmann Machines. Unfortunately, I don't have a background in statistics yet beyond a basic understanding, and a lot of the resources I have been finding online use it heavily. I have an understanding of Hopfield networks, so any answer that could relate the two networks with minimal use of statistics would be greatly appreciated.

So far, every resource I have found has mentioned that RBMs learn the probability distribution of its training patterns, and training involves minimizing (or maximizing) an objective function of the difference between the "true" and "learned" distributions. My question is the following:

What does the "probability distribution" learned by an RBM refer to?

My guess is that for each activation state of the network, it learns a distribution over its training patterns that gives the probability of each training pattern resulting in that activation state, but I haven't found anything to confirm that.

Answer 1

The RBM has a natural dynamics. If you let this dynamics run, it will generate a sequence of activation patterns in its visible (and hidden) nodes. The relative frequencies of these patterns can be described by probabilities. If the RBM has been trained, then the relative frequencies of the activation patterns of the visible nodes will mimic the patterns of the training set.

When the RBM is trained for classification, the visible nodes are separated into input and output nodes, corresponding to the separation of each training item into "sample" $x$ and "label" $y$. One is often not interested in the joint distribution of $(x, y)$, but given a sample $x$, one is interested in the predicted probabilities of the class $y$ (the conditional distribution $p(y|x)$). Letting the RBM dynamics run while fixing the values of the input nodes gives a sequence of output values that should ideally follow this conditional distribution.

For example suppose that each sample $x$ contains three bits and that the class $y$ contains one bit. Then the RBM needs $3+1=4$ visible nodes. Suppose we want to classify the sample 010. We set the four visible nodes to 010$y$, with $y$ random. We start the RBM dynamics, but we always fix the three visible nodes corresponding to $x$ to 010, and we observe the output on the fourth visible node, corresponding to $y$. If, in the training set, the sample 010 always appears with class 1, then also the RBM should output 1 almost always. In general (if the me 010 does not occur in the training set), the output of the RBM for $y$ will be random, according to the RBM's best guess of the probability $p(y|x=010)$.

Answer 2

The probability distribution modeled by an RBM is:

$$P(v) = \sum_h P(v,h)$$

where

$$P(v,h) = \frac{1}{Z} \exp(-E(v,h))$$

where $Z$ is the partition function and $E$ is the energy, defined as

$$-E(v,h) = v^TWh + v^Tb + h^Tc$$

where $W, b, c$ are the model parameters and $h$ represents some unknown latent space.

If you'd like you can explicitly compute $Z = \sum_{v,h} \exp(-E(v,h))$.