Ordinary autoencoder architectures (not variational autoencoders, stacked denoising autoencoders, etc.) seem to only have three layers: the input, the hidden/code, and the output/reconstruction. Are there any examples of papers which used architectures consisting of multiple hidden layers? If not, what are the theoretical justifications for only using one hidden layer in an autoencoder?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Are there any examples of papers which used architectures consisting of multiple hidden layers?
Yes, e.g. look for "deep autoencoders" a.k.a. "stacked autoencoders", such as {1}:
Hugo Larochelle has the video on it: Neural networks [7.6] : Deep learning - deep autoencoder
Geoffrey Hinton also has a video on it: Lecture 15.2 — Deep autoencoders [Neural Networks for Machine Learning]
Examples of deep autoencoders which don't make use of pretraining: http://ufldl.stanford.edu/wiki/index.php/Stacked_Autoencoders
A good way to obtain good parameters for a stacked autoencoder is to use greedy layer-wise training.
E.g., {2} uses a stacked autoencoder with greedy layer-wise training.
Note that one can use autoencoders fancier than feedforward fully connected neural networks, e.g. {3}.
References:
- {1} Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. "Reducing the dimensionality of data with neural networks." science 313, no. 5786 (2006): 504-507. https://scholar.google.com/scholar?hl=en&q=Reducing+the+Dimensionality+of+Data+with+Neural+Networks&btnG=&as_sdt=1%2C22&as_sdtp= ; https://www.cs.toronto.edu/~hinton/science.pdf (~5k citations)
- {2} Heydarzadeh, Mehrdad, Mehrdad Nourani, and Sarah Ostadabbas. "In-bed posture classification using deep autoencoders." In Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th Annual International Conference of the, pp. 3839-3842. IEEE, 2016. https://scholar.google.com/scholar?cluster=16153787462804186587&hl=en&as_sdt=0,22
- {3} Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, Koray Kavukcuoglu. Conditional Image Generation with PixelCNN Decoders. NIPS 2016. https://arxiv.org/abs/1606.05328 ; http://papers.nips.cc/paper/6527-tree-structured-reinforcement-learning-for-sequential-object-localization.pdf
answered Jan 19, 2017 at 18:09
-
$\begingroup$ I'm more interested in examples of deep autoencoders which don't make use of pretraining using RBMs, but are trained end-to-end using backprop with random weight initializations. However, this is also very interesting, and it could be the case that every deep autoencoder makes use of pretraining. $\endgroup$– liangjyCommented Jan 19, 2017 at 18:18
-
$\begingroup$ @liangjy I see, answer edited. To obtain good parameters for a feedforward stacked autoencoder one often uses greedy layer-wise training. $\endgroup$ Commented Jan 20, 2017 at 3:24