4
$\begingroup$

Most examples I found on the internet explain well back-propagation in convolution layer, but only with a single kernel and single input channel.

I do not understand how to do back-propagation for more than one kernel and more than one input channel.

Let's say I have a convolution layer which accepts an input $X$ of size 3x20x20, applies 5 3x3x3 kernel filters $K$ and produces an output $O$ of a size 5x18x18

On a diagram it looks like that (I apologize for my horrible hand-drawing): Forward Pass

During the backward pass the layer receives an error $\frac{\partial E}{\partial O}$ and propagates it back to the previous layer.

As I understand, in order to compute $\frac{\partial O}{\partial X}$ I need to apply 'full' convolution to $\frac{\partial E}{\partial O}$ with a kernels rotated 180°. So, it looks like this:

Backward Pass

The dimensions of $\frac{\partial O}{\partial X}$ should match the dimensions of $X$ (3x20x20), but convolution operation produces an output with a depth equals to the number of kernels (in my case 5).

My question is how a 'full' convolution between $\frac{\partial E}{\partial O}$ 5x18x18 over 5 rotated filters 3x3x3 can produce an output $\frac{\partial O}{\partial X}$ with dimension 3x20x20 ? Is not it that the depth of the output of convolution operation equals to the number of filters ?

$\endgroup$
1
+100
$\begingroup$

I try to explain the dimensions obtained (5x18x18 -> 3x20x20):

  • 5 -> 3 the flipped convolutions are repeated 3 times, but the effects of each of the 5 filter are summed up, exactly as you do in the forward phase. In any case in a convolutional layer it is possible to give any depth in input and any number of filters in output as well. enter image description here

  • 18 -> 20 is given by the full convolution, in which is applied a padding to the input image obtaining then a bigger image as result.

Anyway here the backpropagation in convolution layers is very well explained.

$\endgroup$
  • 1
    $\begingroup$ Could you summarize the link in your answer? Link-only answers are deprecated as links can die over time, making this answer less useful to future visitors. $\endgroup$ – Frans Rodenburg Aug 20 '18 at 9:36
  • $\begingroup$ I wrote that the link explains backpropagation for convolutional layers. Anyway it is not the main part of the answer, which is below. $\endgroup$ – Tonca Aug 20 '18 at 9:45
  • $\begingroup$ @Tonka, I appreciate for your effort to explain it, but I still do not get it. I can only do convolution between two volumes of equal depth. Whether kernels rotated 180° or not, does not change its depth. During the backward pass $dE/dO$ I receive from previous layer has a dimension of 5x18x18, but kernels has dimension 3x3x3. Depth of kernel do not match the depth of $dE/dO$. Regarding the link you have provided, it covers case with a single kernel and single input channel and unfortunately does not answer my question. $\endgroup$ – koryakinp Aug 20 '18 at 16:04
  • $\begingroup$ Remember that convolution is only a 2-dimensional operation. You can try to visualize the whole set of filters as a matrix 3x5 and the convolution as a product. If you're propagating backwards then you need to transpose the matrix and you'll get the dimensions you seek. $\endgroup$ – Tonca Aug 21 '18 at 5:54
  • $\begingroup$ @Tonca, Are you saying that in order to compute $\frac{\partial E}{\partial X}$ I need not only to rotate kernels 180°, but also transpose kernel depth and with kernel number i.e. 5 kernels with depth 3 would become 3 kernels with depth 5 ? $\endgroup$ – koryakinp Aug 21 '18 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.