I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done.

Say you have a basic neural network with a standard input layer, single hidden layer, and output layer with a single neuron, that you are to train with labelled data via backpropagation and gradient descent.

To pre-train, create a stack of two RBMs: one for input layer to hidden layer, and one for hidden layer to output neuron. The RBMs should correspond to your neural network, ie. the first should have the same number of input neurons as your network and same number of hidden neurons as your network, etc.

Take your labelled data and ignore the labels, so you have a set of unlabelled data. Then use that unlabelled data to train the first RBM via the normal RBM training algorithm of forward passes and reconstructions. Using this trained RBM, train the second RBM using the same algorithm.

Instead of randomly generating initial weights for your neural network before carrying out gradient descent, you use the weights found for your stack of RBMs.

My question is: Is this correct? Do I have the correct understanding of how this pre-training works please?


According to Erhan et al. (2010) I am correct. That paper contains info on how to do it with a Deep Belief Network.

To summarise, pre-train a neural network by training a Deep Belief Network (a stack of RBMs). The DBN should have the same topology as your NN. The weights and biases of the DBN are the initial weights and biases one should use when training one's NN using gradient descent.

  • $\begingroup$ Generally, I think it is considered good practice to make answers reasonably standalone, and not just link to somewhere else. (edit: but, that said, welcome to Cross-Correlated :) ) $\endgroup$ – Hugh Perkins Jul 25 '18 at 12:16
  • 1
    $\begingroup$ Sorry, I will fix now. $\endgroup$ – GMSL Jul 25 '18 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.