# What does pre-training mean in deep autoencoder?

I am confused by the term "pre-training". What does it mean in deep autoencoder? And how does it help improving the performance of autoencoder? (I know this term comes from Hinton 2006's paper: "Reducing the dimensionality of Data with Neural Networks".)

An auto encoder is a stack of $$K$$ models of the form $$y^k = \sigma(W^ky^{k-1} + b^k)$$ where $$y^{k-1}$$ is the input to the net and $$y^k$$ is its output. It is then trained to minimize some reconstruction loss, e.g. $$\mathcal{L}(W^1, b^1, \dots, W^k, b^k) = ||y^K - y^0||_2^2.$$ Pretraining now means to optimise some similar objective layer wise first: you first minimize some loss $$\mathcal{L}^k$$, starting out at $$k=1$$ to $$k=K$$.
A popular example is to minimize the layer wise reconstruction: $$\mathcal{L}(k) = ||{W^k}^T\sigma(W^ky^{k-1} + b^k||_2^2,$$ wrt to $$W^k, b^k$$. This means that each auto encoder learns first to auto encode the input to itself.
• Yes, you are. But often only half of the layers is pretrained, and the remaining ones are initialized with the tranpose of the other layer. Eg. $W^1 = {W^{K}}^T$. This only works when the sizes are the same, though. Oct 16, 2014 at 13:37