I know we use pretraining in order to get better performance. but this is a high level explanation. I want to know what exactly happens when we use pre training. does it help the network not to get stuck in local minimas?
If its so why is it said (e.g in Stanford's convnet videos here) that Local minimas are not of that importance that they are in the normal and conventional neural networks, and as it is said be Justin Johnson in their video lectures ?


[Answer to your first question]

Apparently these effects are not fully understood yet (Deep Learning, Goodfellow et al).

It is hypothesised that pretraining will initialise the network in a region which would cause it to approach a different local minima which would otherwise be inaccessible (due to random initialisation). The set of weights learned with pretraining is speculated to have a regularising effect (i.e. improved test error rates).

The second reason, applicable to unsupervised learning, is that features learned in modelling the input are "somehow" useful for a discriminative task like classification.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.