1
$\begingroup$

I'm trying to develop the equations for back propagation for convolutional networks.

Let's say I have a network with one convolutional network (no pooling) with no activation function. The activation function of the layer is

$$f = x\times k$$

where $x$ is the input matrix and $k$ is my convolution kernel. If I derive $f$ by $k$:

$$\frac{df}{dk} = x \times \frac{dk}{dk}$$

If $k$ is a vector in size $N{\times}1$ then $\frac{dk}{dk}$ is a unit matrix $N{\times}N$.

This derivative doesn't fit in size with what I expect and I can't propagate it further. I assume I'm doing something wrong...

$\endgroup$
0
$\begingroup$

I'm not very knowledgeable on convolutional networks but when you apply back propagation to regular neural networks you differentiate the error value of the network with respect to each connections weight.

Where the error value is the difference between the actual network output and the desired output.

In order to back propagate this error value, the activation functions also have to be differentiated; but this is still with respect to connection weights.

So I would be expecting to see your functions f being differentiated with respect to the connection weight values, and not with respect to a convolution kernel k.

Though my lack of knowledge of convolutional networks could be hiding some subtlety of the question.

$\endgroup$
1
$\begingroup$

The derivative is $x$, and in order to calculate that functional derivative, you need to apply calculus of variations. Is very much like calculating $$ \frac{d}{dx}ax = a $$ for any a. To see it in informal terms, take the case of standard derivative for that case: $$ a(x+\epsilon)-ax = a\epsilon $$ for any x. If you divide by $\epsilon$, you are left with a constant. For convolutions is trickier: you take a new kernel, $k+k^{'}$, and calculate the difference, $$(k+k^{'})*f - k*f = k^{'}*f$$ so it is independent of $k$. Here, $k^{'}$ is a perturbation of the kernel, a small variation of its shape. It plays the role of the $\epsilon$. Taking the similarities even further, you could take Fourier transformations and divide it by the Fourier transformation of $k'$, and you would be left with $f$.

Regretfully, I am not aware of any accesible introduction to functional derivatives.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.