7
$\begingroup$

I don't have a strong math background, so I am kind of stuck trying to understand how RBMs work. On wikipedia page about RBMs, I am able to understand the formulas in isolation, however, I'm not sure how these formulas are connected together.

More specifically, isn't there cyclic dependency between energy function, probability functions & sampling process? In the training algorithm, it states to use Gibbs sampling together with other stuff in order to build/train W. However, sampling depends on P, and P by definition depends on E, and E depends on W, so in my noob eyes then this is cyclic dependency.

Could someone please clarify with a short example how I could break this "loop"?

Thanks

http://en.wikipedia.org/wiki/Restricted_Boltzmann_machine

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Not quite sure what you're getting at: P is a function of E and E is a function of W, so forget about E and just say that P is a function of W. The aim of the training is to maximise $\mathbb{E}[\sum_v \log P(v)]$ with respect to $W$. $\endgroup$
    – queenbee
    Commented May 15, 2014 at 21:06

2 Answers 2

Reset to default
5
$\begingroup$

While it seems that the OP is not interested in this question anymore (and, based on the profile information, in Cross Validated, for that matter), I've decided to add some additional information, which is IMHO relevant and hopefully will be useful for the community.

First and foremost, I would like to share an excellent tutorial on deep learning, which contains a whole section dedicated to restricted Boltzmann machines (RBM). This tutorial by LISA Lab team (guided by Youshua Bengio) is available online as well as in several document formats. The RBM section can be found here and in Chapter 9 of the corresponding PDF version of the tutorial. Code samples are presented in Python, with some focus on GPU-enabled Theano deep learning library. Speaking of software, the rbm R package is an example of a few R ecosystem's packages for deep learning: https://github.com/zachmayer/rbm. Extensive collections of references to libraries and other types of software for machine learning, including deep learning and RBM, can be found here, here and here.

Finally, returning from talking about software to the subject of literature and resources, this page contains an extensive list of resources on deep learning, in general, and RBM, in particular.

Improve this answer
$\endgroup$
0
$\begingroup$
  1. First look into this paper in deep.

A. Fischer and C. Igel, "An Introduction to Restricted Boltzmann machines," in Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, ed: Springer, 2012, pp. 14-36.

  1. Then, You may look into Hinton's coursera course website. You will understand proper.

In RBM, neuron states are stochastically decided based on p(v|h) and p(h|v). These two equations are derived from energy equation. For training, maximum likelihood(ML) approximation method is applied.

To avoid the complexity in training using ML method, gradient descent method is applied. This will provide us learning rules. Again here, we need to calculate for huge no of nodes. So Gibbs sampling methods enter into the scene. Gibbs sampling also have problem of convergence.So we use approximation method, which is known as CD method / PCD..etc

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.