# What does pre-training mean in deep autoencoder?

I am confused by the term "pre-training". What does it mean in deep autoencoder? And how does it help improving the performance of autoencoder? (I know this term comes from Hinton 2006's paper: "Reducing the dimensionality of Data with Neural Networks".)

An auto encoder is a stack of $$K$$ models of the form $$y^k = \sigma(W^ky^{k-1} + b^k)$$ where $$y^{k-1}$$ is the input to the net and $$y^k$$ is its output. It is then trained to minimize some reconstruction loss, e.g. $$\mathcal{L}(W^1, b^1, \dots, W^k, b^k) = ||y^K - y^0||_2^2.$$ Pretraining now means to optimise some similar objective layer wise first: you first minimize some loss $$\mathcal{L}^k$$, starting out at $$k=1$$ to $$k=K$$.

A popular example is to minimize the layer wise reconstruction: $$\mathcal{L}(k) = ||{W^k}^T\sigma(W^ky^{k-1} + b^k||_2^2,$$ wrt to $$W^k, b^k$$. This means that each auto encoder learns first to auto encode the input to itself.

Note that this strategy is obsolete nowadays due to non-saturating transfer functions, better understanding of the optimisation problem and GPUs.

• SO, basically it's that I pre-train each layer of autoencoder sequentially? For example, let assume the total number of layers is 5. I pre-train the first layer first, and then use the first layer as an input to second layer, and then pre-train second layer; perform this until all layers are pre-train. Then, fine tune the entire autoencoder. Am I right? Oct 14, 2014 at 16:19
• Yes, you are. But often only half of the layers is pretrained, and the remaining ones are initialized with the tranpose of the other layer. Eg. $W^1 = {W^{K}}^T$. This only works when the sizes are the same, though. Oct 16, 2014 at 13:37

Actually, if you pre-train all the layers to learn the activations of the previous one, I found it may perform sub-optimally during the subsequent fine-tuning. I get a much better performance when I set the last layer during pre-training to try to reconstruct the original input (the one fed to the first layer) instead of the activations of the previous hidden layer. This way the resulted multi-layer autoencoder during fine-tuning will really reconstruct the original image in the final output.

See my post here: How to train and fine-tune fully unsupervised deep neural networks?