I am trying to find a source for which academic paper introduced CNNs in deep learning.

  • $\begingroup$ I am not sure this question is really "on topic", a better venue might be here). That said, I am no expert and my knowledge is essentially the recent Nature paper cited below, but from there it seems LeCun introduced them ca. 1990, with inspiration from Fukushima's Neocognitron 1982. CNNs were introduced prior to the terminology "ConvNet" or "Deep Learning", so what is the "first" paper? (e.g. possibly even on topic for HSM.SE?) $\endgroup$ – GeoMatt22 Oct 26 '16 at 15:50
  • $\begingroup$ en.wikipedia.org/wiki/Convolutional_neural_network#History provides some overview. $\endgroup$ – Franck Dernoncourt Oct 26 '16 at 16:38
  • $\begingroup$ Asking about neural networks is on topic here, as is asking for references pertaining to on topic subject matter. I don't see any problem with this question. $\endgroup$ – gung - Reinstate Monica Oct 26 '16 at 16:39
  • $\begingroup$ OK. I made my comment into an answer. The original question had a "chatty" phrasing which threw me off. Also, from some of the recent migrations, I was not sure if perhaps there was a preference to have "this type" of question (i.e. non-technical) go to AI.SE? $\endgroup$ – GeoMatt22 Oct 26 '16 at 17:03
  • $\begingroup$ Can you clarify whether you mean introducing in the sense of the first to describe or in the sense of providing a basic introduction? $\endgroup$ – mdewey Oct 26 '16 at 17:15

Two recent papers come to mind:

Lecun, Bengio and Hinton, Deep Learning, http://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf

Schmidhuber, Deep Learning in Neural Networks: An Overview, http://people.idsia.ch/~juergen/DeepLearning2July2014.pdf


As mentioned in another answer, the recent Deep Learning Review in Nature has some good discussion of this. I am not in this field, so cannot say how much this is a balanced review vs. just the authors' perspective. However one of the authors, Yann LeCun, seems to be widely acknowledged as essentially the "father of convolutional neural networks" (a.k.a. ConvNets, CNNs). So the Nature review is a reasonable source for this topic, at the very least.

Given this, the reference chosen as the "origin of deep learning in ConvNets" there is essentially this 1990 NIPS paper:

LeCun et al. (1990) Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems.

The review mentions some earlier work on convolution-type layers, but notes that:

This is the first paper on convolutional networks trained by backpropagation for the task of classifying low-resolution images of handwritten digits.

The terms I have bolded lead me to conclude that this paper is a reasonable candidate for the "academic paper that introduced CNNs in deep learning" (i.e. BackProp would seem to be an essential component for the "deep" aspect).

However it is worth noting that in the Nature review the authors mention that the approach was significantly inspired by

Fukushima (1980) Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics.

Finally, as noted in the comments to the OP, Wikipedia has a summary of the history of ConvNets.

  • $\begingroup$ I am not sure if there is a preferred abbreviation for Convolutional Neural Network? I myself prefer "ConvNet" over "CNN". So a question about "CNNs" is not about these, and "CNN++" would be some improved version of a ConvNet, rather than a successor to this! $\endgroup$ – GeoMatt22 Oct 26 '16 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.