I am currently trying some meta-algorithms on training neural networks. Start with a small but expressive enough network for training and after several epochs, initialize a larger neural network with the trained weight in the small neural network. It seems to me that we can gain marginal improvement in the optimization speed in terms of time.

I believe there are already lots of paper in this direction but could not find any paper related possibly because I am using a wrong key word. I wonder where I can find relevant papers, thanks in advance.

  • 1
    $\begingroup$ layerwise pretraining is a good keyword $\endgroup$ – shimao Apr 7 at 1:57

This paper, its bibliography, and the papers citing this paper will be a good coverage of the topic.

Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle. "Greedy Layer-Wise Training of Deep Networks" NIPS 2007.

Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently it was not clear how to train such deep networks, since gradient-based optimization starting from random initialization appears to often get stuck in poor solutions. Hinton et al. recently introduced a greedy layer-wise unsupervised learning algorithm for Deep Belief Networks (DBN), a generative model with many layers of hidden causal variables. In the context of the above optimization problem, we study this algorithm empirically and explore variants to better understand its success and extend it to cases where the inputs are continuous or where the structure of the input distribution is not revealing enough about the variable to be predicted in a supervised task. Our experiments also confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.

Note that layer-wise training of neural networks is a bit of a historical curiosity at this point. Today, deep networks are trained "all at once." This is accomplished by using nicer activations (the ReLU family), nicer initializations, and smarter optimizers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.