0
$\begingroup$

RBM

The energy function for RBM (Restricted Boltzmann Machine) is defined as $$ E(v,h) = -\sum_{i,j} w_{ij} \, v_i \, h_j -\sum_i a_i \, v_i - \sum_i b_i \, h_i $$ with the joint distribution $$ \tag{1} p(v, h) = \frac 1 Z \exp(-E(v,h)) $$

I am thinking why this form of potential function (or energy function) can satisfy the Markov properties of RBM graph?

I've read some relevant posts (e.g., 1 and 2) but they are answered with high level arguments and none of them have convinced me mathematically.

To answer this question, I have read through the proof of Hammersley-Clifford Theorem and I've noticed the potential function constructed in the proof is of the following form: $$ \phi_a(x)=\sum_{b: b\subseteq a}(-1)^{|a\backslash b|} H_b(x) = \sum_{b: b\subseteq a}(-1)^{|a\backslash b|} \log f(x_b, *) $$ ($\phi_a(x)=0$ if $a$ is not a clique)

and the theorem proves the joint probability can be factorized into: $$ \tag{2} f(x)=\exp \sum_{a:a\subseteq V} \phi_a(x) $$ ($V$ are all vertices in the graph)

and this distribution will respect the Markov properties (by (F)=>(G)=>(L)=>(P)=>(F), see the proofs in Chapt16 of Jordan's book draft).

So without considering the normalization (i.e., $\frac1Z$), we can simply compare RHS of equation (1) and (2) and I think they have to be the same form (otherwise no theorem says (P)=>(F)). What I do not get is that they look very similar but the signs do not match.

To be more specific, as for RBM, I let $$ \left\{ \begin{array}{rl} f(v_i,h_j,*) &= \exp(w_{ij}\, v_i \, h_j) > 0\\ f(v_i,*) &= \exp(a_i\, v_i) > 0\\ f(h_j,*) &= \exp(b_j\, h_j) > 0\\ f(\emptyset,*) &= c > 0\\ \end{array} \right. $$ Plugging into equation (2) I can get $$ \begin{align} f(v,h) &= \exp(\sum_{i,j}\phi_{v_i,h_j}(x) + \sum_i \phi_{v_i}(x) + \sum_i \phi_{h_i}(x) + C) \\ &\propto \exp(\sum_{i,j}\phi_{v_i,h_j}(x) + \sum_i \phi_{v_i}(x) + \sum_i \phi_{h_i}(x)) \\ &= \exp(\sum_{i,j}\phi_{v_i,h_j}(x) + \sum_i (-1)^0 a_i\, v_i + \sum_i (-1)^0 b_j\, h_j) \\ &= \exp(\sum_{i,j}\phi_{v_i,h_j}(x) + \sum_i a_i\, v_i + \sum_i b_j\, h_j) \\ &= \exp(\sum_{i,j} ( (-1)^{|2-2|} w_{ij}\, v_i \, h_j + (-1)^{|2-1|} a_i\, v_i + (-1)^{|2-1|} b_j\, h_j ) + \\ & \sum_i a_i\, v_i + \sum_i b_j\, h_j) \\ &= \exp(\sum_{i,j} ( w_{ij}\, v_i \, h_j - a_i\, v_i - b_j\, h_j) + \sum_i a_i\, v_i + \sum_i b_j\, h_j) \\ \end{align} $$ since each vertex generally has more than 1 degree in RBM graph, the last equation is equal to $$ f(v,h) = \exp(\sum_{i,j} w_{ij}\, v_i \, h_j - L \sum_i a_i\, v_i - M \sum_i b_j\, h_j) $$ where $L, M > 0$.

I compare it to equation (1), and it seems the signs and coefficients do not match. Why is it?

$\endgroup$
0
$\begingroup$

Aha, I think by defining these functions instead $$ \left\{ \begin{array}{rl} f(v_i,h_j,*) &= \exp(w_{ij}\, v_i \, h_j) > 0\\ f(v_i,*) &= \exp(-a'_i\, v_i) > 0\\ f(h_j,*) &= \exp(-b'_j\, h_j) > 0\\ f(\emptyset,*) &= c > 0\\ \end{array} \right. $$

I can get to the expected form: $$ \begin{align} f(v,h) &= \exp(\sum_{i,j} w_{ij}\, v_i \, h_j + L\sum_i a'_i\, v_i + M \sum_i b'_j\, h_j) \\ &= \exp(\sum_{i,j} w_{ij}\, v_i \, h_j + \sum_i a_i\, v_i + \sum_i b_j\, h_j) \end{align} $$

But still not sure if this is really the derivation behind RBM energy function.

Please correct me if I am wrong.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.