Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually).

First let me try to explain how I understand backpropagation on a fully connected network.

enter image description here

For example, the derivative of the $Error$ with respect to $W_1$ is the following:

$$ \frac{\partial Error}{\partial W_1} = \frac{\partial Error}{\partial HA_1} \frac{\partial HA_1}{\partial H_1} \frac{\partial H_1}{\partial W_1} $$

The last partial derivative is the most interesting one in this case ... and it is equal to the value of the first input (Single value).

$$ \frac{\partial H_1}{\partial W_1} = I_1 $$


The original question was how does one perform backpropagation on a convolutional layer - for example $$\frac{\partial Error}{\partial W_1} = ?$$

The convolutional layer as described online.

enter image description here

$$ G_1 = V_1W_1 + V_2W_2 + V_4W_3 + V_5W_4 \\ G_2 = V_2W_1 + V_3W_2 + V_5W_3 + V_6W_4 \\ G_3 = V_4W_1 + V_5W_2 + V_7W_3 + V_8W_4 \\ G_4 = V_5W_1 + V_6W_2 + V_8W_3 + V_9W_4 $$

Notice that there are groups of pixels that share the same weights ($W$s), so I can picture the equations above as follows:

enter image description here

So, applying the chain rule as in the first example, we get the following:

$$ \frac{\partial Error}{\partial W_1} = \frac{\partial Error}{\partial G_1}\frac{\partial G_1}{\partial W_1} + \frac{\partial Error}{\partial G_2}\frac{\partial G_2}{\partial W_1} + \frac{\partial Error}{\partial G_3}\frac{\partial G_3}{\partial W_1} + \frac{\partial Error}{\partial G_4}\frac{\partial G_4}{\partial W_1} $$

And the derivatives of interest ... $$ \frac{\partial G_1}{\partial W_1} = V_1 \\ \frac{\partial G_2}{\partial W_1} = V_2 \\ \frac{\partial G_3}{\partial W_1} = V_4 \\ \frac{\partial G_4}{\partial W_1} = V_5 $$

That's it! Chain rule all the way.

I don't like to answer my own question - so if you leave some feedback or tell me I am wrong - I 'll give you the credit.

  • 2
    $\begingroup$ Backpropagation is the reverse mode (accumukation) of automatic differentiation (a.k.a. algorithmic differentiation) applied to neural networks. $\endgroup$ – Mark L. Stone Feb 3 '18 at 3:01
  • 3
    $\begingroup$ I think it can be summarized this way: backprop of a convolution is again a convolution. $\endgroup$ – Maxim Feb 4 '18 at 11:13
  • 8
    $\begingroup$ You're not merely permitted to answer your own question, but if you've found the answer on your own, you are encouraged to answer your own question. Moreover, this question has had no answers for several months, so I think that it's more than fair for you to answer it. $\endgroup$ – Sycorax Aug 18 '18 at 16:54

Could you not simply say that the backpropagation on a convolutional layer is the sum of the backpropagation on each part, sliding window, of the image/tensor that the convolution covers?

This is important as it connects to the fact that the weights are shared over multiple pixels and thus weights should reflect general local features of the images independently from their location.

| cite | improve this answer | |
  • $\begingroup$ I think that could be true if there is no overlap - so window and stride are equal. $\endgroup$ – Edv Beq Jul 23 at 14:38
  • $\begingroup$ It's true in general. If you consider the overlapping parts of the images. In other words if the same pixel is in position x1,y1 in window (image part) 1 and in position x2,y2 in window (image part) 2, it will affect the gradient of a different weight for each image part. Still, the total gradient for each weight will be the sum of the gradients of the weight for each window/image part. $\endgroup$ – Dimitri Ognibene Jul 23 at 14:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.