I am trying to understand how CNN works. I want to use them in object recognition task. I thouhgt that CNN is unsupervised networks. My main question is how can I implement the back propagation without using image targets (pre-defined labels), in order to calculate the error function which will affect the weights of the network.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ I would like to see an answer for this question =) $\endgroup$– jjepsuomiCommented Sep 25, 2014 at 11:56
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
The derivation involves considerations for the fact that CNNs use weight sharing as opposed to feed-forward networks. During both forward and back-propagation you will use convolutions where the weights and the activations will be the functions in the convolution equation.
The pooling layers do not do any learning themselves. Their function is to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network.
During forward propagation, a P by P pooling block is reduced to a single value i.e. value of the “winning unit”. To keep track of the “winning unit” its index noted during the forward pass and used for gradient routing during backpropagation.
During backpropagation, the gradients in the convolutional layers are calculated and the backward pass to the pooling layer then involves assigning the “winning unit” the gradient value from the convolutional layer as the index was noted prior during the forward pass.
Gradient routing is done in the following ways:
Max-pooling - the error is just assigned to where it comes from - the “winning unit” because other units in the previous layer’s pooling blocks did not contribute to it hence all the other assigned values of zero
Average pooling - the error is multiplied by 1 / (P by P) and assigned to the whole pooling block (all units get this same value).
Read a more comprehensive breakdown on the whole backpropagation procedure here
$\begingroup$
$\endgroup$
4
CNN are typically supervised. So you need training data, and you forward propagate the training images through the network, then back propagate the training labels, to update the weights.
There are unsupervised neural networks, for example Geoffrey Hinton's stacked boltmann machines, creating deep belief networks. You can also stack auto-encoders, which can also perform unsupervised learning.
In both cases, the goal of the unsupervised learning is that the result of forward propagation followed by backward propagation is the original image.
answered Apr 29, 2015 at 13:44
-
$\begingroup$ Not a bad answer, but Boltzmann machines don't do any "backpropagation" at all; they use contrastive divergence. Autoencoders explicitly use the source image as the target: the goal is for forward propagation (alone) to reproduce the original image; backpropagation propagates the error gradients to correct prediction error. $\endgroup$– Neil GCommented Apr 29, 2015 at 13:56
-
$\begingroup$ Boltzmann machine's propagate 1s and 0s randomly, according to the probabilities of their weights, in both directions, forwards and backwards. If you define backward propagation to mean propagating loss gradients, then, by your definition, boltzmann machines dont do that. $\endgroup$ Commented Apr 29, 2015 at 14:02
-
$\begingroup$ That's usually what people mean by "backpropagation": en.wikipedia.org/wiki/Backpropagation $\endgroup$– Neil GCommented Apr 29, 2015 at 14:11
-
$\begingroup$
$\endgroup$
Backpropagation is basically a clever way of expressing and calculating the gradient of a cost function, which depends on known target labels and its corresponding inputs. That is the original formulation of CNNs. They are step beyond traditional MLPs, where features are not designed, but automatically learned from data, so that the recognition rate is maximized (with respect to known labels).
If I understood you question properly, I would reformulate it like: is it possible to train a discriminative model without labels (just data, or partially labelled data?).
What I networks does is defined by the loss (or energy) function it minimizes. What sort of gradient based optimization technique you apply, depends on that energy function. This topic is analyzed in detail in the following references: