Pre-training in deep convolutional neural network?

Question

Have anyone seen any literature on pre-training in deep convolutional neural network? I have only seen unsupervised pre-training in autoencoder or restricted boltzman machines.

Answer 1

I'm not sure if this exactly answers your question, but from what I understand the reason you don't see people pretraining (I mean this in an unsupervised pretraining sense) conv nets is because there have been various innovations in purely supervised training that have rendered unsupervised pretraining unnecessary (for now, who knows what problems and issues the future will hold?).

One of the main innovations was moving away from sigmoidal (sigmoid, tanh) activation units, which can saturate/have regions of near flat curvature and thus very little gradient gets propagated backwards, so learning is incredibly slow if not completely halted for all practical intents and purposes. The Glorot, Bordes and Bengio article Deep Sparse Rectifier Neural Networks used rectified linear units (ReLUs) as activation functions in lieu of the traditional sigmoidal units. The ReLUs have the following form: $f(x) = \max(0, x)$. Notice that they are unbounded and for the positive part, has constant gradient 1.

The Glorot, Bordes and Bengio article used ReLUs for multilayer perceptrons and not Conv Nets. A previous article What is the best Multi-Stage Architecture for Object Recognition by Jarret and others from Yann LeCun's NYU group used rectifying nonlinearities but for the sigmoidal units, so they had activation functions of the form $f(x) = |\tanh(x)|$, etc. Both articles observed that using rectifying nonlinearities seems to close much of the gap between purely supervised methods and unsupervised pretrained methods.

Another innovation is that we have figured out much better initializations for deep networks. Using the idea of standardizing variance across the layers of a network, good rules of thumb have been established over the years. One of the first, most popular ones was by Glorot and Bengio Understanding the Difficulty of Training Deep Feedforward Networks which provided a way to initialize deep nets under a linear activation hypothesis and later on Delving Deep Into Rectifiers by a group of Microsoft Research team members which modify the Glorot and Bengio weight initialization to account for the rectifying nonlinearities. The weight initialization is a big deal for extremely deep nets. For a 30 layer conv net, the MSR weight initialization performed much better than the Glorot weight initialization. Keep in mind that the Glorot paper came out in 2010 and the MSR paper came out in 2015.

I am not sure if the ImageNet Classification with Deep Convolutional Neural Networks paper by Alex Krizhevsky, Ilya Sutskever and Geoff Hinton was the first to use ReLUs for conv nets, but it had the biggest impact. In this paper we see that ReLUs for conv nets speeds up learning, as evidenced by one of their CIFAR-10 graphs which shows that ReLU conv nets can achieve lower training error rates faster than non-ReLU conv nets. These ReLUs do not suffer from the vanishing gradient/saturating sigmoidal issues and can be used to train much deeper nets. One of the other big innovations has been the use of Dropout training, a stochastic noise injection or model averaging technique (depending on your point of view) which allows us to train deeper, bigger neural networks longer without overfitting as much.

And the conv net innovation continued at a blistering pace, nearly all of the methods using ReLUs (or some modification like PReLUs from Microsoft Research), Dropout, and purely supervised training (SGD + Momentum, possibly some adaptive learning rate techniques like RMSProp or ADAGrad).

So as of now, a lot of the top performing conv nets seem to be of a purely supervised nature. That's not to say that unsupervised pretraining or using unsupervised techniques may not be important in the future. But some incredibly deep conv nets have been trained, have matched or surpassed human level performance on very rich datasets, just using supervised training. In fact I believe the latest Microsoft Research submission to the ImageNet 2015 contest had 150 layers. That is not a typo. 150.

If you want to use unsupervised pretraining for conv nets, I think you would be best finding a task where "standard" supervised training of conv nets doesn't perform so well and try unsupervised pretraining.

Unlike natural language modeling, it seems to be hard to find an unsupervised task that helps a corresponding supervised task when it comes to image data. But if you look around the Internet enough, you see some of the pioneers of deep learning (Yoshua Bengio, Yann LeCun to name a few) talk about how important they think unsupervised learning is and will be.

Answer 2

As can be understood from the above answers, pre-training was 'fashioned out' when multiple things happened. However, I do want to distill my understanding of it:

1. Long time ago in 2010, everyone cared about pre-training. Here is a great paper on the subject that I did not see brought up.
2. Slightly before before Alex Krizhevsky, Ilya Sutskever and Geoff Hinton published their imagenet paper, people still believed features mattered but were focused mostly on unsupervised learning and even self taught learning to manufacture those features.
3. It is not hard to see why - the building blocks of neural networks at the time were not as robust and converged very slowly to useful features. Many times they even failed spectacularly. Pre training was useful when you had ample data you could get a good initialization for SGD.
4. When relu was brought up, networks converged faster. When leaky relu and more recent solutions were brought up, neural nets became more robust machines when it comes to converging to a viable result. I highly recommend that you play with an excellent neural networks demo this talented googler wrote, you will see what I am talking about.
5. Getting to our main point, that is not to say that some form of Pre-training is not important in deep learning. If you want to get state of the art results you have to perform pre-processing of the data (ZCA for example) and properly choose the initial weights - this is a very good paper on the subject.

So you see, pre-training changed in form to pre-processing and weights initialization but remained in function and it became more elegant.

As a final note, machine learning is very fashionable. I am personally betting like Andrew Ng that unsupervised and self-taught learning will be dominant in the future so don't make this a religion :)

Answer 3

There are some papers but not as much as autoencoders or RBMs. I think the reason is the time line of NN. Stacked RBM and autoencoder are introduced at 2006 and 2007, respectively. After employment of ReLU at 2009 unsupervised learning is partially abandoned (when there is enough data to learn in direct supervised learning). Even though Convolution net (or LeNet) is invented at 1989, it couldn't trained as deep structure till 2012 which is after popularization of direct supervised learning with ReLU. So researchers, I guess, have trained it mostly by using direct supervised learning.