# What is pretraining and how do you pretrain a neural network?

I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because backpropagation is slow, and also that I can get stuck in local optima and not learn certain features.

What I don't understand is how we pretrain a network and what specifically we do to pretrain. For example, if we're given a stack of restricted Boltzmann Machines, how would we pretrain this network?

• Unless you are in a setting with only a few labeled and many unlabeled samples, pretraining is considered obsolete. If that is not the case case, using a rectifier transfer function $f(x) = \max(x, 0)$ and advanced optimisers (rmsprop, adadelta, adam) works equally well for deep neural networks. – bayerj Apr 22 '15 at 18:25
• Yeah, I'm working under an assumption that there's a large amount of unlabeled samples and few to no labeled samples. – Michael Yousef Apr 22 '15 at 18:29