# What is pre training a neural network?

Well the question says it all.

What is meant by "pre training a neural network"? Can someone explain in pure simple English?
I can't seem to find any resources related to it. It would be great if someone can point me to them.

• www.deeplearningbook.org has a good introduction to the topic. – mnagaraj Jan 4 '17 at 9:38

The usual way of training a network:

You want to train a neural network to perform a task (e.g. classification) on a data set (e.g. a set of images). You start training by initializing the weights randomly. As soon as you start training, the weights are changed in order to perform the task with less mistakes (i.e. optimization). Once you're satisfied with the training results you save the weights of your network somewhere.

You are now interested in training a network to perform a new task (e.g. object detection) on a different data set (e.g. images too but not the same as the ones you used before). Instead of repeating what you did for the first network and start from training with randomly initialized weights, you can use the weights you saved from the previous network as the initial weight values for your new experiment. Initializing the weights this way is referred to as using a pre-trained network. The first network is your pre-trained network. The second one is the network you are fine-tuning.

The idea behind pre-training is that random initialization is...well...random, the values of the weights have nothing to do with the task you're trying to solve. Why should a set of values be any better than another set? But how else would you initialize the weights? If you knew how to initialize them properly for the task, you might as well set them to the optimal values (slightly exaggerated). No need to train anything. You have the optimal solution to your problem. Pre-training gives the network a head start. As if it has seen the data before.

What to watch out for when pre-training:

The first task used in pre-training the network can be the same as the fine-tuning stage. The datasets used for pre-training vs. fine-tuning can also be the same, but can also be different. It's really interesting to see how pre-training on a different task and different dataset can still be transferred to a new dataset and new task that are slightly different. Using a pre-trained network generally makes sense if both tasks or both datasets have something in common. The bigger the gap, the less effective pre-training will be. It makes little sense to pre-train a network for image classification by training it on financial data first. In this case there's too much disconnect between the pre-training and fine-tuning stages.

Pretraining / fine-tuning works as follows:

1. You have machine learning model $m$.
2. Pre-training: You have a dataset $A$ on which you train $m$.
3. You have a dataset $B$. Before you start training the model, you initialize some of the parameters of $m$ with the model which is trained on $A$.
4. Fine-tuning: You train $m$ on $B$.

This is one form of transfer learning. So you can transfer some of the knowledge obtained from dataset $A$ to dataset $B$. See my Machine Learning Glossary for this and more terms explained in very few words.

• Side note: people might use the term fine-tune only if you (don't) freeze the weights. I'm not sure if there is a consistent use of this term – Martin Thoma May 30 '18 at 13:07

The two answers above explains well. Just want to add one subtle thing regarding the pre-training for Deep Belief Nets (DBN). The pre-training for DBN is unsupervised learning (i.e. w/o labeled data) and the training afterwards is supervised learning (i.e. w/. labeled data).