9
$\begingroup$

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually).

First let me try to explain how I understand backpropagation on a fully connected network.

enter image description here

For example, the derivative of the $Error$ with respect to $W_1$ is the following:

$$ \frac{\partial Error}{\partial W_1} = \frac{\partial Error}{\partial HA_1} \frac{\partial HA_1}{\partial H_1} \frac{\partial H_1}{\partial W_1} $$

The last partial derivative is the most interesting one in this case ... and it is equal to the value of the first input (Single value).

$$ \frac{\partial H_1}{\partial W_1} = I_1 $$

EDITED

The original question was how does one perform backpropagation on a convolutional layer - for example $$\frac{\partial Error}{\partial W_1} = ?$$

The convolutional layer as described online.

enter image description here

$$ G_1 = V_1W_1 + V_2W_2 + V_4W_3 + V_5W_4 \\ G_2 = V_2W_1 + V_3W_2 + V_5W_3 + V_6W_4 \\ G_3 = V_4W_1 + V_5W_2 + V_7W_3 + V_8W_4 \\ G_4 = V_5W_1 + V_6W_2 + V_8W_3 + V_9W_4 $$

Notice that there are groups of pixels that share the same weights ($W$s), so I can picture the equations above as follows:

enter image description here

So, applying the chain rule as in the first example, we get the following:

$$ \frac{\partial Error}{\partial W_1} = \frac{\partial Error}{\partial G_1}\frac{\partial G_1}{\partial W_1} + \frac{\partial Error}{\partial G_2}\frac{\partial G_2}{\partial W_1} + \frac{\partial Error}{\partial G_3}\frac{\partial G_3}{\partial W_1} + \frac{\partial Error}{\partial G_4}\frac{\partial G_4}{\partial W_1} $$

And the derivatives of interest ... $$ \frac{\partial G_1}{\partial W_1} = V_1 \\ \frac{\partial G_2}{\partial W_1} = V_2 \\ \frac{\partial G_3}{\partial W_1} = V_4 \\ \frac{\partial G_4}{\partial W_1} = V_5 $$

That's it! Chain rule all the way.

I don't like to answer my own question - so if you leave some feedback or tell me I am wrong - I 'll give you the credit.

$\endgroup$
  • 1
    $\begingroup$ Backpropagation is the reverse mode (accumukation) of automatic differentiation (a.k.a. algorithmic differentiation) applied to neural networks. $\endgroup$ – Mark L. Stone Feb 3 '18 at 3:01
  • 1
    $\begingroup$ I think it can be summarized this way: backprop of a convolution is again a convolution. $\endgroup$ – Maxim Feb 4 '18 at 11:13
  • 4
    $\begingroup$ You're not merely permitted to answer your own question, but if you've found the answer on your own, you are encouraged to answer your own question. Moreover, this question has had no answers for several months, so I think that it's more than fair for you to answer it. $\endgroup$ – Sycorax Aug 18 '18 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.