6
$\begingroup$

The Hammersley-Clifford Theorem tells us that the distribution of a RBM must be Gibbs since it is Markov Random Field, but how to prove that its energy function must be of the form:

$$E = -\sum_{i,j} w_{ij} \, v_i \, h_j -\sum_i \alpha_i \, v_i - \sum_i \beta_i \, h_i$$

$\endgroup$

1 Answer 1

3
$\begingroup$

You only need to verify that the graphical model that represents a RBM fulfills the definition of a MRF as given in the document you refer to. See here for a picture.

Then it is warranted that you can write it as a factorized product of positive functions defined on cliques that cover all the nodes and edges of G. Now, the theorem does not says that the decomposition is unique. It is mathematically convenient and meaningful to find a factorial decomposition through a energy function like that one. What you can do is very that this energy function gives you a valid factorial decomposition of the probability distribution in terms of positive functions which depend on the maximal cliques of the graph.

In this case the (maximal) cliques are rather trivial: just the pairs of nodes (again, take a look at the graph). Each pair has a corresponding term in the sum. When raising the exponential to $E$, the sum turns to a product of positive functions (the exponential is a positive function), where each function depends only on two variables corresponding to each of the cliques.

$\endgroup$
4
  • $\begingroup$ Thanks - but given the decomposition is not unique, why is this particular function so "mathematically convenient and meaningful"? $\endgroup$
    – mchen
    Commented Jun 11, 2013 at 19:46
  • $\begingroup$ It's convenient because it's linear. The linearity is particularly useful when you want to find gradients with respect to the coefficients ($w$, $\alpha$, and $\beta$ in your notation). The gradients are useful for optimizing the coefficients. It's meaningful because the it has a relationship to the inverse covariances among nodes. $\endgroup$ Commented Jun 11, 2013 at 20:05
  • $\begingroup$ Actually, the relationship to the covariance may only apply if the nodes have Gaussian distributions--I'll have to think about it for a minute. But @juampa's point about factorizing is a very good one: finding a way to factorize the distribution makes just about everything in RBMs easier; if you know $h$, it's trivial write down the distribution of $v$ and vice versa. That wouldn't be so straightforward if the parameterization were different. $\endgroup$ Commented Jun 11, 2013 at 20:11
  • $\begingroup$ To appreciate why it is so convenient take a look at the paper "A fast learning algorithm for deep belief nets" by Hinton et al. It is not only because you only need to perform trivial matrix multiplications to calculate the state of the nodes, but also because you get rid of explaining away effects which facilitates convergence to a meaningful representation of data. For more insight at an intuitive level, you may watch this talk (youtube.com/watch?v=AyzOUbkUf3M) $\endgroup$
    – jpmuc
    Commented Jun 12, 2013 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.