Summary of Pre-Training a Neural Network with Stacks of RBMs

Question

I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done.

Say you have a basic neural network with a standard input layer, single hidden layer, and output layer with a single neuron, that you are to train with labelled data via backpropagation and gradient descent.

To pre-train, create a stack of two RBMs: one for input layer to hidden layer, and one for hidden layer to output neuron. The RBMs should correspond to your neural network, ie. the first should have the same number of input neurons as your network and same number of hidden neurons as your network, etc.

Take your labelled data and ignore the labels, so you have a set of unlabelled data. Then use that unlabelled data to train the first RBM via the normal RBM training algorithm of forward passes and reconstructions. Using this trained RBM, train the second RBM using the same algorithm.

Instead of randomly generating initial weights for your neural network before carrying out gradient descent, you use the weights found for your stack of RBMs.

My question is: Is this correct? Do I have the correct understanding of how this pre-training works please?

Answer 1

According to Erhan et al. (2010) I am correct. That paper contains info on how to do it with a Deep Belief Network.

To summarise, pre-train a neural network by training a Deep Belief Network (a stack of RBMs). The DBN should have the same topology as your NN. The weights and biases of the DBN are the initial weights and biases one should use when training one's NN using gradient descent.