Recently, I was asked about how to pre-train a deep neural network with unlabeled data, meaning, instead of initializing the model weight with small random numbers, we set initial weight from a pretrained model (with unlabeled data).

Well, intuitively, I kinda get it, it probably helps with the vanishing gradient issue and shorten the training time when there are not too much labeled data available. But still, I don't really know how it is done, how can you train a neural network with unlabeled data? Is it something like SOM or Boltzmann machine?

Has anybody heard about this? If yes, can you provide some links to sources or papers. I am curious. Greatly appreciate!


1 Answer 1


A common approach to using unlabeled data to pre-train a network is by first contructing a denoising auto-encoder. Corrupted versions of the unlabeled data are then reconstructed by the autoencoder (source task), after which the encoding part of the network can be used for a regression or classification task (target task).

The reason this sometimes benefits the target task is because the early layers of the network are already trained to recognize useful features of the input data. It also acts as a regularizer, constraining the parameter space for the target task.

Have a look here for example. Figure 6 demonstrates the regularizing ability of this approach.

Linked article:

  • Erhan D. et al. (2010): Why Does Unsupervised Pre-training Help Deep Learning?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.