12
$\begingroup$

I searched on Google, Wikipedia, Google scholar, and more, but I could not find the origin of Autoencoders. Perhaps it's one of those concepts that evolved very gradually, and it's impossible to trace back a clear starting point, but still I would like to find some kind of summary of the main steps of their development.

The chapter about autoencoders in Ian Goodfellow, Yoshua Bengio and Aaron Courville's Deep Learning book says:

The idea of autoencoders has been part of the historical landscape of neuralnetworks for decades (LeCun, 1987; Bourlard and Kamp, 1988; Hinton and Zemel,1994). Traditionally, autoencoders were used for dimensionality reduction or feature learning.

This presentation by Pascal Vincent says:

Denoising using classical autoencoders was actually introduced much earlier (LeCun, 1987; Gallinari et al., 1987), as an alternative to Hopfield networks (Hopfield, 1982).

This seems to imply that "classical autoencoders" existed before that: LeCun and Gallinari used them but did not invent them. I see no trace of "classical autoencoders" earlier than 1987.

Any ideas?

$\endgroup$
9
$\begingroup$

According to the history provided in Schmidhuber, "Deep learning in neural networks: an overview," Neural Networks (2015), auto-encoders were proposed as a method for unsupervised pre-training in Ballard, "Modular learning in neural networks," Proceedings AAAI (1987). It's not clear if that's the first time auto-encoders were used, however; it's just the first time that they were used for the purpose of pre-training ANNs.

As the introduction to the Schmidhuber article makes clear, it's somewhat difficult to attribute all of the ideas used in ANNs because the literature is diverse and terminology has evolved over time.

$\endgroup$
  • 1
    $\begingroup$ The paper written by Ballard , has completely different terminologies , and there is not even a sniff of the Autoencoder concept in its entirety. Maybe AE does not have any origins paper. $\endgroup$ – abunickabhi Sep 21 '18 at 10:45
  • 2
    $\begingroup$ The Schmudhuber paper emphasizes that the terminology has changed over time and different people have rediscovered the same topics over and over. It’s not surprising that the author doesn’t use the word “auto encoder” $\endgroup$ – Sycorax Sep 21 '18 at 13:11
4
$\begingroup$

The paper below talks about autoencoder indirectly and dates back to 1986.(which is a year earlier than the paper by Ballard in 1987)

D.E. Rumelhart, G.E. Hinton, and R.J. Williams, "Learning internal representations by error propagation." , Parallel Distributed Processing. Vol 1: Foundations. MIT Press, Cambridge, MA, 1986.

The paper basically describes a novel kind of feedforward network at that time , and its mathematical formalism.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.