4
$\begingroup$

I am confused by the term "pre-training". What does it mean in deep autoencoder? And how does it help improving the performance of autoencoder? (I know this term comes from Hinton 2006's paper: "Reducing the dimensionality of Data with Neural Networks".)

$\endgroup$
4
$\begingroup$

An auto encoder is a stack of $K$ models of the form $$ y^k = \sigma(W^ky^{k-1} + b^k) $$ where $y^{k-1}$ is the input to the net and $y^k$ is its output. It is then trained to minimize some reconstruction loss, e.g. $$ \mathcal{L}(W^1, b^1, \dots, W^k, b^k) = ||y^K - y^0||_2^2. $$ Pretraining now means to optimise some similar objective layer wise first: you first minimize some loss $\mathcal{L}^k$, starting out at $k=1$ to $k=K$.

A popular example is to minimize the layer wise reconstruction: $$ \mathcal{L}(k) = ||{W^k}^T\sigma(W^ky^{k-1} + b^k||_2^2, $$ wrt to $W^k, b^k$. This means that each auto encoder learns first to auto encode the input to itself.

Note that this strategy is obsolete nowadays due to non-saturating transfer functions, better understanding of the optimisation problem and GPUs.

$\endgroup$
  • $\begingroup$ SO, basically it's that I pre-train each layer of autoencoder sequentially? For example, let assume the total number of layers is 5. I pre-train the first layer first, and then use the first layer as an input to second layer, and then pre-train second layer; perform this until all layers are pre-train. Then, fine tune the entire autoencoder. Am I right? $\endgroup$ – RockTheStar Oct 14 '14 at 16:19
  • 1
    $\begingroup$ Yes, you are. But often only half of the layers is pretrained, and the remaining ones are initialized with the tranpose of the other layer. Eg. $W^1 = {W^{K}}^T$. This only works when the sizes are the same, though. $\endgroup$ – bayerj Oct 16 '14 at 13:37
1
$\begingroup$

Actually, if you pre-train all the layers to learn the activations of the previous one, I found it may perform sub-optimally during the subsequent fine-tuning. I get a much better performance when I set the last layer during pre-training to try to reconstruct the original input (the one fed to the first layer) instead of the activations of the previous hidden layer. This way the resulted multi-layer autoencoder during fine-tuning will really reconstruct the original image in the final output.

See my post here: How to train and fine-tune fully unsupervised deep neural networks?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.