Based on my research on convolution neural networks, every other layer in such a network has a subsampling operation, in which the resolution of the image is reduced so as to improve generalization of the network. So, a CNN could consist of an alternation of convolution and subsampling layers. However, when using backpropagation to train a convolutional neural network, I don't quite understand how one would train a convolutional layer. When you are training a convolution layer, don't you need the weights of the next layer to calculate delta? Based on my understanding of backprop, the equation to find delta of a hidden layer is as follows:

Neuron.Delta = Neuron.Output * (1 - Neuron.Output) * ErrorFactor

But, to find ErrorFactor, you need the weights of the connections between the current layer and the next one. And if the next layer is a subsampling layer, then there will not be any weights to use to calculate delta. My current solution to this problem is to simply look at the next layer's weights to calculate the delta of the current layer. So, if layer 1 is a convolution layer, layer 2 is a subsampling layer, and layer 3 is a convolution layer, I would look at the weights connecting layers 2 and 3 to calculate the delta at layer 1. Is this a correct understanding of how to train a convolution neural network?


If I understand you correctly, the question is how to train the net if you have pooling layers? Well, the weights in pooling layers are not that different from the ones in "normal" layers. Imagine you have a max pooling layer with grid size 3x3. Imagine further that for a given training example, pixel number 5 (that is, in position (2,2) ) has had the max value in forward propagation, i.e. its value has been passed through the max pooling layer. When doing backprop for that sample, the weight between your pixel number 5 and the output of the pooling is simply one, while for the other eight pixels it is zero. And since the max pooling does not do any further transformation, the error used is that from the layer that came after the max pooling layer. For a more mathematical formulation, there is a nice website: http://andrew.gibiansky.com/blog/machine-learning/convolutional-neural-networks/


Back-propagation is "just" a nice rick to combine the chain rule with dynamic programming so that you get an efficient method to calculate the gradient of the network with respect to its parameters.

Whether you can calculate the gradient does not depend on whether there are weights. You still have to calculate the derivatives for the subsampling layers, even though there are no weights. It's easy to imagine this if you consider that subsampling is just a very special kind of transfer function.

  • $\begingroup$ Can you explain how dynamic programming enters into it? That seems incorrect to me. $\endgroup$ – David J. Harris Jan 12 '14 at 7:05
  • $\begingroup$ Dynamic programming in the computer science terminology (not optimization) where it corresponds to caching function calls. See en.wikipedia.org/wiki/Memoization. $\endgroup$ – bayerj Jan 17 '14 at 15:36
  • $\begingroup$ So is this basically the reverse (accumulation) mode of Automatic Differentiation? $\endgroup$ – Mark L. Stone Aug 25 '15 at 23:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.