2
$\begingroup$

I was reading this interesting article on convolutional neural networks. It showed this image, explaining that for every receptive field of 5x5 pixels/neurons, a value for a hidden value is calculated.

receptive field to neuron

We can think of max-pooling as a way for the network to ask whether a given feature is found anywhere in a region of the image. It then throws away the exact positional information.

So max-pooling is applied.

enter image description here

With multiple convolutional layers, it looks something like this:

enter image description here

But my question is, this whole architecture could be build with perceptrons, right?

For every convolutional layer, one perceptron is needed, with layers:

input_size = 5x5;
hidden_size = 10; e.g.
output_size = 1;

Then for every receptive field in the original image, the 5x5 area is inputted into a perceptron to output the value of a neuron in the hidden layer. So basically doing this for every receptive field: enter image description here

So the same perceptron is used 24x24 amount of times to construct the hidden layer, because:

is that we're going to use the same weights and bias for each of the 24×24 hidden neurons.

And this works for the hidden layer to the pooling layer as well, input_size = 2x2; output_size = 1;. And in the case of a max-pool layer, it's just a max() function on an array.

and then finally:

The final layer of connections in the network is a fully-connected layer. That is, this layer connects every neuron from the max-pooled layer to every one of the 10 output neurons.

which is a perceptron again.

So my final architecture looks like this:

-> 1 perceptron for every convolutional layer/feature map
-> run this perceptron for every receptive field to create feature map
-> 1 perceptron for every pooling layer
-> run this perceptron for every field in the feature map to create a pooling layer
-> finally input the values of the pooling layer in a regular ALL to ALL perceptron

Or am I overseeing something? Or is this already how they are programmed?

$\endgroup$
  • $\begingroup$ Take a look at the pattern recognition book by Duda and Hart. $\endgroup$ – Michael Chernick Mar 8 '17 at 14:00
2
$\begingroup$

Much depends on how strict you interpret the definition of perceptron. Typically the perceptron refers to the precursor of what is called a neuron in neural networks: it is a linear binary classification function that only allows as output a 1 or a 0.

According to this definition, the answer to your question would be 'no' since convolutional networks require more than a binary output and the max-pooling layer is not linear.

Much confusion comes from the name of the successor to perceptrons: multi-layer perceptrons. MLP's no longer have linear activation functions and are no longer restricted to binary outputs.

MLP's are also different from convolutional networks however in the sense that all layers are fully-connected.

Update: Can MLP's be used to construct a fully convolutional network? You could argue that you could use MLP's with one layer to construct parts of the convolutional network, but by definition MLP's have multiple layers, so again the answer would be 'no'.

$\endgroup$
  • $\begingroup$ Thats what I mean, ill edit it soon: the MLPs can be used to CONSTRUCT a CNN if I read everything correcty? $\endgroup$ – Thomas W Mar 8 '17 at 17:18
  • $\begingroup$ updated my answer to reflect this question $\endgroup$ – GR4 Mar 8 '17 at 17:27
  • $\begingroup$ I like your answer, but what im saying is using a perceptron to calculate the hidden layer from a receptive field, which does not oupit binary data even when its just 1 layer. Im using a javascript library synaptic.js and depending on the Squash function, you wont have a binary output. I still cant see how it is not possible to create a CNN by the use (the same) of a perceptron for every detail that you want to pick up of the original input. $\endgroup$ – Thomas W Mar 8 '17 at 17:30
  • $\begingroup$ I just had a quick look at synaptic.js and the squash function. Note that they speak of neurons and not perceptrons. The point is that synaptic.js doesn't use the strict original definition of a perceptron, but uses neurons as building blocks to which non-linear activation functions can be applied. Under those circumstances: yes, you can reproduce the CNN with those building blocks. In fact, that is all a CNN is. $\endgroup$ – GR4 Mar 8 '17 at 17:39
1
$\begingroup$

Each convolutional layer is essentially (quite like) a perceptron since it's just computing linear sums of the inputs, which are then passed through some non-linearity (like a sigmoid or ReLU). However, note that a defining feature of the convolutional layer is that it applies each linear filter at every position of the image (or at every position of the previous layer). This is not true for a perceptron, which would have to have a separate filter for each position. This also has important consequences for the way the network is trained. In a convolutional net, you have to learn each filter only once. Using perceptrons, you'd have to learn the same filter separately for each position in the image (which is a harder learning problem).

I guess you could say that this means each convolutional filter is simply a set of many identical but shifted perceptrons, but at that point I think you'd be stretching the definitions a bit. So in short, the two are clearly related but (at least in my view) the convolution operation is a defining characteristic that sets the convolutional net apart from perceptrons.

$\endgroup$
  • $\begingroup$ Hmm, but why is a different perceptron needed? Every hidden neuron in the convolutional layer fires when a certain pattern is found in the 5x5 receptive field right? Why would the perceptron need to be different for every field, its trying to find the same partern right? $\endgroup$ – Thomas W Mar 8 '17 at 17:56
  • $\begingroup$ Each hidden neuron fires when the same pattern is found at a slightly different position. You could say "that means it's the same perceptron repeated at every position in the image". But that's not how a perceptron is normally trained. A perceptron learns a fixed weight for each entry of the input vector, not a set of weights that can be shifted and repeated across the image. This is what I mean by stretching definitions: if you take a perceptron and "upgrade" it with these capabilities, I don't think you can call it a perceptron any more. It would be like calling a car a "repeated motorbike". $\endgroup$ – Ruben van Bergen Mar 10 '17 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.