4
$\begingroup$

So we have "several RBMs"

Deep Belief Network

A deep belief network is obtained by stacking several RBMs on top of each
other. The hidden layer of the RBM at layer `i` becomes the input of the
RBM at layer `i+1`. The first layer RBM gets as input the input of the
network, and the hidden layer of the last RBM represents the output. When
used for classification, the DBN is treated as a MLP, by adding a logistic
regression layer on top.

but why more RBM's don't means better? How to determine best number of layers?

Seems related: Deep belief network performs worse than a simple MLP

$\endgroup$

1 Answer 1

1
$\begingroup$

I haven't worked the proof myself, but according to Hinton et al (2006) more RBMs should in fact always be better* as another RBM always increases the lower bound on the probability that the training data can be reproduced.

*The exception to this is when an RBM has as many or more visible units as the RBM before it has hidden layers, as then the chances of simply learning the identify function go up. For example, if you have an input of dimension 1000, then you should always get an increase in predictive performance if your next layers are 700, 500, 200, etc. up to the point at which the output of the last RBM matches the "true" dimensionality of variation in the data, which is a question of experimentation or expertise.

$\endgroup$
2
  • $\begingroup$ @patrik : Can you verify the exception claim that you made through some good reference??? $\endgroup$ Jan 10, 2018 at 17:15
  • $\begingroup$ @churchurnaan I doubt what he says. There is no such a concept like identity function in RBM. I guess he is talking about auto-encoder. $\endgroup$ May 30, 2018 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.