7
$\begingroup$

Most examples I found on the internet explain well back-propagation in convolution layer, but only with a single kernel and single input channel.

I do not understand how to do back-propagation for more than one kernel and more than one input channel.

Let's say I have a convolution layer which accepts an input $X$ of size 3x20x20, applies 5 3x3x3 kernel filters $K$ and produces an output $O$ of a size 5x18x18

On a diagram it looks like that (I apologize for my horrible hand-drawing): Forward Pass

During the backward pass the layer receives an error $\frac{\partial E}{\partial O}$ and propagates it back to the previous layer.

As I understand, in order to compute $\frac{\partial O}{\partial X}$ I need to apply 'full' convolution to $\frac{\partial E}{\partial O}$ with a kernels rotated 180°. So, it looks like this:

Backward Pass

The dimensions of $\frac{\partial O}{\partial X}$ should match the dimensions of $X$ (3x20x20), but convolution operation produces an output with a depth equals to the number of kernels (in my case 5).

My question is how a 'full' convolution between $\frac{\partial E}{\partial O}$ 5x18x18 over 5 rotated filters 3x3x3 can produce an output $\frac{\partial O}{\partial X}$ with dimension 3x20x20 ? Is not it that the depth of the output of convolution operation equals to the number of filters ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
+100
$\begingroup$

I try to explain the dimensions obtained (5x18x18 -> 3x20x20):

  • 5 -> 3 the flipped convolutions are repeated 3 times, but the effects of each of the 5 filter are summed up, exactly as you do in the forward phase. In any case in a convolutional layer it is possible to give any depth in input and any number of filters in output as well. enter image description here

  • 18 -> 20 is given by the full convolution, in which is applied a padding to the input image obtaining then a bigger image as result.

Anyway here the backpropagation in convolution layers is very well explained.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ Could you summarize the link in your answer? Link-only answers are deprecated as links can die over time, making this answer less useful to future visitors. $\endgroup$
    – Frans Rodenburg
    Commented Aug 20, 2018 at 9:36
  • $\begingroup$ I wrote that the link explains backpropagation for convolutional layers. Anyway it is not the main part of the answer, which is below. $\endgroup$
    – Tonca
    Commented Aug 20, 2018 at 9:45
  • $\begingroup$ @Tonka, I appreciate for your effort to explain it, but I still do not get it. I can only do convolution between two volumes of equal depth. Whether kernels rotated 180° or not, does not change its depth. During the backward pass $dE/dO$ I receive from previous layer has a dimension of 5x18x18, but kernels has dimension 3x3x3. Depth of kernel do not match the depth of $dE/dO$. Regarding the link you have provided, it covers case with a single kernel and single input channel and unfortunately does not answer my question. $\endgroup$
    – koryakinp
    Commented Aug 20, 2018 at 16:04
  • $\begingroup$ Remember that convolution is only a 2-dimensional operation. You can try to visualize the whole set of filters as a matrix 3x5 and the convolution as a product. If you're propagating backwards then you need to transpose the matrix and you'll get the dimensions you seek. $\endgroup$
    – Tonca
    Commented Aug 21, 2018 at 5:54
  • 1
    $\begingroup$ @Tonca, Are you saying that in order to compute $\frac{\partial E}{\partial X}$ I need not only to rotate kernels 180°, but also transpose kernel depth and with kernel number i.e. 5 kernels with depth 3 would become 3 kernels with depth 5 ? $\endgroup$
    – koryakinp
    Commented Aug 21, 2018 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.