I am confused by the term "pre-training". What does it mean in deep autoencoder? And how does it help improving the performance of autoencoder? (I know this term comes from Hinton 2006's paper: "Reducing the dimensionality of Data with Neural Networks".)
$\begingroup$
$\endgroup$
1
asked Oct 14, 2014 at 5:59
-
1$\begingroup$ See also: What is pre training a neural netwok $\endgroup$– Martin ThomaApr 10, 2017 at 12:36
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
An auto encoder is a stack of $K$ models of the form $$ y^k = \sigma(W^ky^{k-1} + b^k) $$ where $y^{k-1}$ is the input to the net and $y^k$ is its output. It is then trained to minimize some reconstruction loss, e.g. $$ \mathcal{L}(W^1, b^1, \dots, W^k, b^k) = ||y^K - y^0||_2^2. $$ Pretraining now means to optimise some similar objective layer wise first: you first minimize some loss $\mathcal{L}^k$, starting out at $k=1$ to $k=K$.
A popular example is to minimize the layer wise reconstruction: $$ \mathcal{L}(k) = ||{W^k}^T\sigma(W^ky^{k-1} + b^k||_2^2, $$ wrt to $W^k, b^k$. This means that each auto encoder learns first to auto encode the input to itself.
Note that this strategy is obsolete nowadays due to non-saturating transfer functions, better understanding of the optimisation problem and GPUs.
-
$\begingroup$ SO, basically it's that I pre-train each layer of autoencoder sequentially? For example, let assume the total number of layers is 5. I pre-train the first layer first, and then use the first layer as an input to second layer, and then pre-train second layer; perform this until all layers are pre-train. Then, fine tune the entire autoencoder. Am I right? $\endgroup$ Oct 14, 2014 at 16:19
-
1$\begingroup$ Yes, you are. But often only half of the layers is pretrained, and the remaining ones are initialized with the tranpose of the other layer. Eg. $W^1 = {W^{K}}^T$. This only works when the sizes are the same, though. $\endgroup$– bayerjOct 16, 2014 at 13:37
$\begingroup$
$\endgroup$
Actually, if you pre-train all the layers to learn the activations of the previous one, I found it may perform sub-optimally during the subsequent fine-tuning. I get a much better performance when I set the last layer during pre-training to try to reconstruct the original input (the one fed to the first layer) instead of the activations of the previous hidden layer. This way the resulted multi-layer autoencoder during fine-tuning will really reconstruct the original image in the final output.
See my post here: How to train and fine-tune fully unsupervised deep neural networks?
