2
$\begingroup$

I was studying Deep Belief Network (DBN) and have questions.

1) According to the definition of DBN, DBN is formed by stacking RBM on top of each other such that the hidden layer in a lower layer becomes the input layer in the above layer. However, when I read the papers by Geoff Hinton (for example, "a fast learning algorithm for deep belief nets"), his DBN doesn't seem to have multiple RBMs. His DBN has only one RBM (i.e., undirected Markov Random Field with two layers) which sits on top while the other layers have only directional edges. I am confused by this difference.

Are these two architectures essentially same? When the term, "DBN" is used in literatures, does it refer to Geoff Hinton's DBN? Or does it refer to a general class of multi-layer architectures with one or multiple RBMs.

2) Following up the above question, does anybody know why Geoff Hinton's DBN has undirected RBM (called associative memory) on top? What role does it play? What would happen if, instead of undirected edges, directed edges are used on top?

Thanks,

Improve this question
$\endgroup$
1
  • $\begingroup$ By the way, here is a Coursera course taught be Professor Hinton that goes through neural networks in detail. coursera.org/course/neuralnets $\endgroup$
    – John Yetter
    Commented Apr 29, 2014 at 20:12

1 Answer 1

Reset to default
5
$\begingroup$

Geoffrey Hinton definitely uses multiple layers of RBM's in some cases. I believe he coined the term Deep Belief Network.

The point of the associative memory is that those layers, which are unsupervised, discover more optimal features for the feed forward layer that follows than just treating the input as a bag of bits.

In the case of image recognition, the first RBM layer might start to detect edges. The second would start to detect larger features. Finally, the larger scale features would become apparent. In the more specific case of images of faces, this leads to a set of faces that act similarly to "eigen faces". This allows the next layers (i.e. the feed forward neural network) to have an input that is essentially a combination of these eigen faces, which make for better features for the feed forward network.

There are some additional techniques, such as training with drop out, that Hinton uses to increase predictive power.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have read that in the last few years, use of techniques such as use of rectified linear units and training with drop out has rendered the RBM layer obsolete. According to FastML these techniques provide better results, faster, and adding RBM layers no longer improves things past that point. $\endgroup$
    – John Yetter
    Commented May 2, 2014 at 17:07
  • 1
    $\begingroup$ Note however that this is only the case when you have enough labeled data to train on. While it is true that ReLUs offer much better training behaviour when using backpropagation, you still need the labeled training data. If you do not have that, RBMs/DBNs are still a good choice. Especially because ReLUs can be applied to RBMs as well. $\endgroup$
    – nemo
    Commented Jul 6, 2014 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.