2
$\begingroup$

I'm trying to understand why pre-training of a deep neural network improves classification performance. By initialising the network weights from a pre-training stage, and then training on data for classification, the performance is usually better than if the weights were initialised randomly.

But why wouldn't random initialisation eventually find this same minimum of the loss function? The loss function uses the same data in both, and both approaches should converge to the same global minimum, right?

If anybody has any intuition about this then it would be helpful, thanks!

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ This question might be useful: stats.stackexchange.com/questions/163600/…. Long story short: your intuition is correct. Top performing conv nets, surpassing human level on object recognition tasks, are now trained from scratch with randomized weight initialization rules with no need for pre-training. $\endgroup$
    – Indie AI
    Commented Feb 28, 2016 at 16:31

1 Answer 1

Reset to default
2
$\begingroup$

This question has been studied extensively in the literature. Pre-training finds a good spot of weights in the error surface. Intuitively, were finding a good set of weights to compressing the input data in the pre-training phase. Ideally, this compressed representation is good for solving generic tasks.

For the record, no one pre-trains deep neural networks anymore. They simply drop some connections and train the entire network with backpropagation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What about if you didn't have very much training data for my classes? Is it still not common to use pre-training these days? For example with object recognition, if you have some new objects which are not present in something like ImageNet, and you only have a few images per object, then would pre-training still be necessary? $\endgroup$
    – Karnivaurus
    Commented Feb 28, 2016 at 19:27
  • $\begingroup$ The premise that pre-training works with small datasets is False. To learn an effective latent representation of variables, we must have enough data in the first place. Pre-training rarely helps with small datasets. Remember, pre-training is used to first extract a good representation of the data. $\endgroup$
    – user46925
    Commented Feb 28, 2016 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.