Good tutorial for Restricted Boltzmann Machines (RBM)

Question

I’m studying the Restricted Boltzmann Machine (RBM) and am having some issues understanding log likelihood calculations with respect to the parameters of the RBM. Even though a lot of research papers on RBM have been published, there are no detailed steps of the derivatives. After searching online I was able to find them in this document:

• Fischer, A., & Igel, C. (2012). An Introduction to Restricted Boltzmann Machines. In L. Alvarez et al. (Eds.): CIARP, LNCS 7441, pp. 14–36, Springer-Verlag: Berlin-Heidelberg. (pdf)

However, the details of this document are too advanced for me. Can somebody point me towards a good tutorial / set of lecture notes about RBM?

---

Edit: @David, the confusing section is shown below (equation 29 in page 26):

\begin{align} \frac{\partial\ln\mathcal{L}(\theta|v)}{\partial w_{ij}} &= -\sum_h p(h|v)\frac{\partial E(v, h)}{\partial w_{ij}} + \sum_{v,h} p(v,h)\frac{\partial E(v,h)}{\partial w_{ij}} \\[5pt] &= \sum_h p(h|v)h_iv_j - \sum_v p(v) \sum_h p(h|v)h_iv_j \\[5pt] &= \color{orange}{\boxed{\color{black}{p(H_i=1|v)}}}v_j - \sum_v p(v) \color{orange}{\boxed{\color{black}{p(H_i=1|v)}}}v_j\; . \tag{29} \end{align}

Answer 1

I know it is a little late, but maybe it helps. To obtain the first term of your equation, it takes these steps: \begin{align} \sum_{\mathbf{h}} p(\mathbf{h} | \mathbf{v})h_iv_j &= v_j \sum_{h_1}...\sum_{h_i}...\sum_{h_n} p(h_1,...,h_i,...h_n | \mathbf{v}) h_i \\[5pt] &= v_j \sum_{h_i} \sum_{\mathbf{h_{\_ i}}}p(h_i, \mathbf{h_{\_i}} | \mathbf{v}) h_i \end{align} We have assumed that conditional independence between the hidden units, given the visible units, exists. Thus we can factorize the conditional joint probability distribution for the hidden states. \begin{align} &= v_j \sum_{h_i} \sum_{\mathbf{h_{\_ i}}} p(h_i | \mathbf{v}) h_i \: p(\mathbf{h_{\_ i}}|\mathbf{v}) \\[5pt] &= v_j \sum_{h_i} p(h_i | \mathbf{v}) h_i \: \sum_{\mathbf{h_{\_ i}}} p(\mathbf{h_{\_ i}}|\mathbf{v}) \end{align} The last term equals $1$, since we are summing over all states. Thus what is left, is the first term. Since $h_i$ only takes states $1$ and $0$ we end up with: $$ \hspace{-25mm}= v_j \: p(H_i = 1 | \mathbf{v}) $$

Answer 2

1. There is a decent tutorial of RBMs on the deeplearning site.

2. This blog post (Introduction to Restricted Boltzmann Machines) is written in simpler language and explains the basics of RBMS really well:

3. Also, maybe the best reference is Geoff Hinton's Neural Networks course on Coursea:

   I'm not sure if you can access the class and videos after the end of the class though.

Answer 3

The left orange box gives you the expected value of the energy gradient over all hidden configurations given that some visible vector is clamped on the visible units (the expectation over the data since it uses a sample from your training set). The term itself is the product of (1) the probability of seeing a particular hidden unit i on given that some vector v is clamped on the visible units and (2) the state of a particular visible unit j.

The right orange box is the same thing as the left one, except you're doing what's in the left orange box for every possible visible configuration instead of just the one that's clamped on the visible units (the expectation over the model since nothing is clamped on the visible units).

Answer 4

Chapter 5 of Hugo Larochelle's course on machine learning (video) is the best introduction I've found so far.

The derivative of the loss function is not derived in these lectures but it's not hard to do it (I can post a scan of my calculations if needed, but it's really not that hard). I'm still looking for a good textbook covering this topic but mainly there are only articles. There is a good overview of the articles in chapter 20 of Bengio's Deep Learning Book.