How to compute the free energy of a RBM given its energy? The free energy of a Restricted Boltzman Machine is defined thusly: 
$$F(v)=-\ln\bigg(\sum_he^{-E(v,h)}\bigg)$$
What I don't understand is how to compute this $\sum_h$? If I know $E(v,h)$, how can I compute $F(v)$? 
I have seen several definitions of $F(v)$ for a binary-binary RBM, but I have not been able to work it out from $E(v,h)$ of the RBM. 
And can I do the same with Gaussian visible units (changing $E(v,h)$), of course)? And what about ReLU units? 
 A: I have a quick answer about the case of binary-binary RBMs.
Suppose we have $J$ units in the hidden layer, and we can use $h_1$ to $h_J$ to denote these units. When $v$ is fixed, for each combination of the values of $h_1$ to $h_J$ (which is a sequence of 0's and 1's), we have an energy $E(v,h)$.
So if we list all the possible combinations of values of $h_1$ to $h_J$, calculate the energy, add $e^{-E(v,h)}$ together, we can get the sum $\sum_{h}e^{-E(v,h)}$.
The summation here means summing all the possible values of $h$.
A: If the joint distribution for Gaussian - Bernouli RBM is $$E(v,h)=1/2(v-b)^T(v-b)-c^Th-v^TWh$$
Then Free Energy $F(v)$ would be :
 $$F(v)=-v^Tb+1/2(v^Tv+b^Tb)-\sum_jlog(1+\exp(c+W^Tv))_j$$
A: The edge configuration of an RBM means you can calculate the free energy given some $\mathbf{v}$ efficiently without resorting to brute force summation, which for many practical problems is the difference between being tractable and not.
\begin{align}
F(\mathbf{v}) &= -\log\sum_{\mathbf{h}}\exp\left(\mathbf{a}^T\mathbf{v} + \mathbf{b}^T\mathbf{h} + \mathbf{v}^T\mathbf{W}\mathbf{h}\right)
\end{align}
The $\exp$ term under the sum fully factorises in terms of the components $h_j$ of $\mathbf{h}$, i.e. it's just a product of terms where each term has at most one component of $\mathbf{h}$. This is true only because there are no edges between hidden units, i.e. the $h_j$ are conditionally independent given $\mathbf{v}$.
This factorisation allows us to push the $\sum_\mathbf{h}$ inside the product and avoid summing over all $\mathbf{h}$ and rather calculate something like $\prod_j \sum_{h_j} (\ldots)$, which is far easier. If you write it out the result for binary RBMs is
$$
F(\mathbf{v}) = -\mathbf{a}^T\mathbf{v} - \sum_j \log \left(\mathbf{1} + \exp\left(\mathbf{b} + \mathbf{W}^T\mathbf{v}\right)\right)_j
$$
which is quick. This is an application of the more general sum-product algorithm which exploits conditional independence properties to do efficient inference in graphical models.
A python implementation of this free energy calculation is here: https://github.com/lisa-lab/DeepLearningTutorials/blob/master/code/rbm.py#L127
