0
$\begingroup$

From Goodfellow et al.'s Deep Learning book:

Several key concepts arose during (...) the 1980s that remain central to today’s deep learning. One of these concepts is that of distributed representation (Hinton et al., 1986). This is the idea that each input to a system should be represented by many features, and each feature should be involved in the representation of many possible inputs. For example, suppose we have a vision system that can recognize cars, trucks, and birds and these objects can each be red, green, or blue. One way of representing these inputs would be to have a separate neuron or hidden unit that activates for each of the nine possible combinations: red truck, red car, red bird, green truck, and so on. This requires nine different neurons, and each neuron must independently learn the concept of color and object identity. One way to improve on this situation is to use a distributed representation, with three neurons describing the color and three neurons describing the object identity. This requires only six neurons total instead of nine (...)

However, as far as I'm aware, pre-existing models such as Rosenblatt's perceptron [1] and Fukushima's neocognitron [2] made no assumptions regarding a one-to-one relationship between the input and the first layer of neurons.

My question is: was the concept of each neuron being descriptive of a feature of the data, as opposed to representing a single possible combination of the input, introduced by Hinton?


[1] F. Rosenblatt, "The perceptron: A probabilistic model for information storage and organization in the brain", Psychological Review, 1958

[2] K. Fukushima, "A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position", Biological Cybernetics, 1980

$\endgroup$

1 Answer 1

3
$\begingroup$

I don't think Goodfellow et al. are saying that Hinton introduced the concept of distributed representations, just that he co-authored a chapter in the PDP book that provided a good description of distributed representation. I re-read the PDP book recently (my first neural network book from back in 1989) and skimmed the chapter, and it doesn't seem to me to be claiming any priority on the idea. I think it has earlier roots in biological neuroscience and the concept had been around for a while. It is what a lot of neural systems do, so Hinton was unlikely to be the only one in the group (and outside it) that noticed. The PDP book is describing the body of work by a large group of researchers working on different aspects of parallel distributed processing, and most of it is of a summary/tutorial nature.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.