1
$\begingroup$

I realize this is a copy of this question: How to compute the free energy of a RBM given its energy? however I am unable to comment on the best answer to ask that person to elaborate.

We start with:

$$ F(v) = -b^Tv - log\sum_{h} \prod_{j=1}^{M} e^{c_jh_j + v^TW_jh_j} $$

In this tutorial http://deeplearning.net/tutorial/rbm.html it simplifies to:

$$ F(v) = -b^Tv - \sum_{j=1}^{M}log\sum_{h_j}e^{c_jh_j + v^TW_jh_j} $$

And we end up with:

$$ F(v) = -b^Tv - \sum_{j=1}^{M}log(1 + e^{c_j + v^TW_j}) $$

In the first step, what are the prerequisites I need to know to understand why the product can be moved outside the sum?

In the second step, where does the 1 come from? Where does the $ h_j $ go?

$\endgroup$
3
$\begingroup$

Answering my own question since I figured it out.

First, it helps if you write down the full h.

In simpler terms and only displaying the relevant part:

$$ \sum_{h=\{0, 1\}^M} \prod_{j=1}^M e^{q_jh_j} $$

Suppose we have M=2, h takes in the values {(0, 0), (0, 1), (1, 0), (1, 1)} so then this would be:

$$ e^{q_0 0}e^{q_1 0} + e^{q_0 0}e^{q_1 1} + e^{q_0 1}e^{q_1 0} + e^{q_0 1}e^{q_1 1} $$

Which can be factored into:

$$ (e^{q_0 0} + e^{q_0 1})(e^{q_1 0} + e^{q_1 1}) $$

And this works in general, i.e.:

$$ \sum_{h=\{1,..,K\}^M} \prod_{j=1}^M f_j(h_j) = \prod_{j=1}^M \sum_{h_j=\{1,..,K\}} f_j(h_j) $$

So in the original equation we would get:

$$ F(v) = -b^Tv - log \prod_{j=1}^M \sum_{h_j=\{0,1\}} e^{(c_j + v^TW_j)h_j} $$

$$ F(v) = -b^Tv - \sum_{j=1}^M log \sum_{h_j=\{0,1\}} e^{(c_j + v^TW_j)h_j} $$

$$ F(v) = -b^Tv - \sum_{j=1}^M log (e^{(c_j + v^TW_j)0} + e^{(c_j + v^TW_j)1})$$

$$ F(v) = -b^Tv - \sum_{j=1}^M log (1 + e^{c_j + v^TW_j})$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.