I've been learning about stacked autoencoders, but wasn't entirely sure how to train them. From what I understand, given layers $h_1,h_2,...,h_n$, we greedily train as follows

For every h in hidden_layers:
    let prev_hs be all the previously trained hidden layers
    let transformed = prev_hs(input_x)
    train h'(h(transformed)) ~ transformed
    delete h'
    add h to list of previously trained hidden layers

Then for fine tuning, we do

Initialize new hidden layers h1', h2',...,hn' with size hn, h(n-1),..., h2, h1
Train hn'(...(h2'(h1'(hn(...(h2(h1(input_x)))))))) ~ input_x

Is this correct, and if not, what does the training procedure look like?


1 Answer 1


In the first part, the training of the new layer is on the output of the previous layer (transformed in your code), not x. Besides that I am not sure fine tuning is necessary. Also, I think you should keep the decoding layers from the training part. You can combine them to create a full autoencoder, or discard them when building a classifier

  • $\begingroup$ Thanks, that was a typo and I've just fixed it. How would it be possible to use the decoding layers? If h1'(h1(x)) ~ x and h2'(h2(h1(x))) ~ x, how would you use h1'? $\endgroup$
    – hyperdo
    Oct 21, 2017 at 1:46
  • $\begingroup$ h2'(h2(h1(x))) ~ h1(x) because the second layer tries to represent the output of the first layer. So I believe you should stack the decoders in reverse order to retrieve x $\endgroup$
    – Moti Cohen
    Oct 21, 2017 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.