23
$\begingroup$

I searched on Google, Wikipedia, Google scholar, and more, but I could not find the origin of Autoencoders. Perhaps it's one of those concepts that evolved very gradually, and it's impossible to trace back a clear starting point, but still I would like to find some kind of summary of the main steps of their development.

The chapter about autoencoders in Ian Goodfellow, Yoshua Bengio and Aaron Courville's Deep Learning book says:

The idea of autoencoders has been part of the historical landscape of neuralnetworks for decades (LeCun, 1987; Bourlard and Kamp, 1988; Hinton and Zemel,1994). Traditionally, autoencoders were used for dimensionality reduction or feature learning.

This presentation by Pascal Vincent says:

Denoising using classical autoencoders was actually introduced much earlier (LeCun, 1987; Gallinari et al., 1987), as an alternative to Hopfield networks (Hopfield, 1982).

This seems to imply that "classical autoencoders" existed before that: LeCun and Gallinari used them but did not invent them. I see no trace of "classical autoencoders" earlier than 1987.

Any ideas?

Improve this question
$\endgroup$
0

4 Answers 4

Reset to default
10
$\begingroup$

According to the history provided in Schmidhuber, "Deep learning in neural networks: an overview," Neural Networks (2015), auto-encoders were proposed as a method for unsupervised pre-training in Ballard, "Modular learning in neural networks," Proceedings AAAI (1987). It's not clear if that's the first time auto-encoders were used, however; it's just the first time that they were used for the purpose of pre-training ANNs.

As the introduction to the Schmidhuber article makes clear, it's somewhat difficult to attribute all of the ideas used in ANNs because the literature is diverse and terminology has evolved over time.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ The paper written by Ballard , has completely different terminologies , and there is not even a sniff of the Autoencoder concept in its entirety. Maybe AE does not have any origins paper. $\endgroup$
    – abunickabhi
    Sep 21, 2018 at 10:45
  • 2
    $\begingroup$ The Schmudhuber paper emphasizes that the terminology has changed over time and different people have rediscovered the same topics over and over. It’s not surprising that the author doesn’t use the word “auto encoder” $\endgroup$
    – Sycorax ♦
    Sep 21, 2018 at 13:11
8
$\begingroup$

The paper below talks about autoencoder indirectly and dates back to 1986.(which is a year earlier than the paper by Ballard in 1987)

D.E. Rumelhart, G.E. Hinton, and R.J. Williams, "Learning internal representations by error propagation." , Parallel Distributed Processing. Vol 1: Foundations. MIT Press, Cambridge, MA, 1986.

The paper basically describes a novel kind of feedforward network at that time , and its mathematical formalism.

Improve this answer
$\endgroup$
1
$\begingroup$

Reviving this thread - In "Neurocomputing" by Robert Hecht-Nielsen @ 1990 there is reference to a 1986 paper by Cottrell/Munro/Zipser that outlines use of a neural network that has the architecture of an autoencoder, and is trained on the identity function, for compression and reconstruction of image data. The term "autoencoder" isn't mentioned, but that's what it is. There is no mention of using it for anomaly detection either. Seems like that usage was discovered later.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It's best to include the full citations of the papers you're referencing so that others can locate them. Is this the paper? Cottrell, G. W., P. Munro, and D. Zipser. "Image compression by back-propagation." ICS report 8702 (1987). Or does it cite another paper? Please edit to clarify. $\endgroup$
    – Sycorax ♦
    May 17, 2022 at 14:39
0
$\begingroup$

The first clear autoencoder presentation featuring a feedforward, multilayer neural network with a bottleneck layer was presented by Kramer in 1991 (full text at https://people.engr.tamu.edu/rgutier/web_courses/cpsc636_s10/kramer1991nonlinearPCA.pdf). He discusses dimensionality reduction and feature extraction and applications such as noise filtering, anomaly detection, and input estimation. Variational autoencoders, referred to as "robust autoassociative neural networks", were anticipated by by Kramer, 1992 (https://www.sciencedirect.com/science/article/abs/pii/009813549280051A?via%3Dihub). It wasn't until 15 years later that Hinton popularized autoencoders for dimensionality reduction, in 2006.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I would add that the Cotrell Munroe Zipser paper (which I found here is a great paper, far before its time. But the three-layer network is really doing principal components analysis (PCA), not capable of nonlinear encoding and decoding. The five-layer network (which was "deep learning" in that era) that Kramer originally described is required to get nonlinear encoding and decoding functions. $\endgroup$
    – Mark Kramer
    Feb 23 at 12:49
  • $\begingroup$ In case you are the author of the papers you reference in the answer, I think you should make an explicit disclosure. See stats.stackexchange.com/help/behavior (the last paragraph). Even though your user name matches the name on the papers, it could also be a coincidence or a pseudonym (CrossValidated doesn't enforce real names). $\endgroup$
    – Igor F.
    Feb 23 at 15:09
  • $\begingroup$ @IgorF. thanks for that, you’re right. I should have disclosed that I am the author of those papers, so I can’t help be a bit biased. But I would note in my defense that the earlier papers, prior to 1990, never explicitly define encoder-decoder functions in the architecture descriptions. Maybe you could read between the lines but I have difficulty seeing an encoder-decoder architecture in those papers. $\endgroup$
    – Mark Kramer
    Feb 24 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.