4
$\begingroup$

Ordinary autoencoder architectures (not variational autoencoders, stacked denoising autoencoders, etc.) seem to only have three layers: the input, the hidden/code, and the output/reconstruction. Are there any examples of papers which used architectures consisting of multiple hidden layers? If not, what are the theoretical justifications for only using one hidden layer in an autoencoder?

$\endgroup$
8
$\begingroup$

Are there any examples of papers which used architectures consisting of multiple hidden layers?

Yes, e.g. look for "deep autoencoders" a.k.a. "stacked autoencoders", such as {1}:

enter image description here

Hugo Larochelle has the video on it: Neural networks [7.6] : Deep learning - deep autoencoder

Geoffrey Hinton also has a video on it: Lecture 15.2 — Deep autoencoders [Neural Networks for Machine Learning]


Examples of deep autoencoders which don't make use of pretraining: http://ufldl.stanford.edu/wiki/index.php/Stacked_Autoencoders

A good way to obtain good parameters for a stacked autoencoder is to use greedy layer-wise training.

E.g., {2} uses a stacked autoencoder with greedy layer-wise training.

Note that one can use autoencoders fancier than feedforward fully connected neural networks, e.g. {3}.


References:

$\endgroup$
  • $\begingroup$ I'm more interested in examples of deep autoencoders which don't make use of pretraining using RBMs, but are trained end-to-end using backprop with random weight initializations. However, this is also very interesting, and it could be the case that every deep autoencoder makes use of pretraining. $\endgroup$ – liangjy Jan 19 '17 at 18:18
  • $\begingroup$ @liangjy I see, answer edited. To obtain good parameters for a feedforward stacked autoencoder one often uses greedy layer-wise training. $\endgroup$ – Franck Dernoncourt Jan 20 '17 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.