When pre-training deep neural networks layer by layer, is it normal to pre-train the layers -which haven't been pre-trained by unsupervised training- by using supervised training before we train the whole network using supervised training? Should we pre-train the classification layers alone, keeping the unsupervisedly trained layers locked, before we unlock them and train the whole network by supervised training?

I can imagine that in the initial stage of supervised learning the features might get altered a lot, due to the fact that the last layer hasn't even started to converge yet when we would perform supervised learning without pre-training the classification layers. This would then radically alter the unsupervisedly learned features based on faulty classification layers which haven't converged yet.

On the other hand, when we would pre-train the classification layers using supervised learning, we might get stuck in a bad local optimum. Because the classification layers have already converged, it is likely that supervised training on the whole network will also have its local optimum nearby.

Is it a good idea to perform pre-training on the classification layers alone, before performing supervised training on the whole network?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.