1
$\begingroup$

There are no shortage of RBM literatures, but they are not easy to follow for beginners of RBM. Some example will help a lot. Example either in MATLAB or a specific example with numbers (like this link : http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/)


Note: this example (http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/) is good, but still lack of details of steps.

I ran this code several times, and each time got different result. The last line print(r.run_visible(user)) gives either [0 1] or [0 0] or [1 0] or [1 1]. What does this mean?

Why does RBM not yield consistent results ?

$\endgroup$
5
$\begingroup$

Not sure if you are looking for a clearer introduction to RBM or just an example.

If you prefer to have a look at an actual implementation, I have implemented the RBM in Matlab because I found existing implementations to lack modularity for my own needs. You can find it here: https://github.com/pixelou/nnbox/blob/master/networks/RBM.m (pretrain method starts at line 119). I cannot guaranty that it is bug free though :-)

For a comprehensive introduction to Restricted boltzmann machines, you can have a look at Training restricted Boltzmann machines: An introduction from Asja Fischer & Christian Igel, this is the clearest paper in terms of proofs and structure. But I found it a bit hard to follow on the first read so you can instead split your reading into several stages:

1 - consider the problem of learning a model (RBM) as a minimization of the distance between the parametric pdf of the RBM and the underlying dataset distribution.

2 - compute the gradient of the distance so that you can do gradient descent

I have detailed the computation here: How to derive the gradient formula for the Maximum Likelihood in RBM?

3 - notice that you cannot compute that expression because one part is intractable :-). Use Contrastive divergence (or any other method but CD is the most popular) to approximate this value. You can read A New Learning Algorithm for Mean Field Boltzmann Machines from Welling & Hinton (2001) which is one of the early publications on CD.

4 - Use that gradient to perform gradient update, adjust with regularization and tricks. For that matter, a Practical Guide to Training Restricted Boltzmann Machines from Hinton (2010) is a must read.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ @bill If this answer is correct, please select it, otherwise be more specific in your question. Thanks $\endgroup$ – pixelou Feb 23 '16 at 15:28
  • $\begingroup$ Hi pixelou Thanks a lot, really appreciate... I am monkeying with your source code, but meet quite some errors/bugs. Could you please provide a version of example that is bug free, i.e., can be run on MATLAB ? // For example, in MNIST_DNN.m line 46, there is "e % LultilayerNet is configured to pretrain layer wise" and apparently "e" is typo. After I deleted this letter "e", ran MNIST_DNN.m again, I got more errors : Error using RBM/vis2hid Too many input arguments. Error in RBM/compute (line 112) Y = self.vis2hid(X, [], false); ....... $\endgroup$ – Bill Ancalagon the black Feb 25 '16 at 22:47
  • $\begingroup$ Hi bill. Sorry you had to crash-test the example file. I have pushed an updated version, see if it works for you now. $\endgroup$ – pixelou Feb 28 '16 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.