How to update filter weights in CNN?

Question

I am trying to build a Convolutional Neural Network (CNN) from scratch to learn its basics. However, I haven't found any information about how the weights from the kernels get updated in each iteration using backpropagation.

The only explanation I found on the internet was this one but I'm not sure if that is right or if I didn't implement it correctly in MATLAB.

I'm using "same" convolution and $3\times3\times1$ kernels. As far as I understand, all the positions of the kernels appear at all the locations of the images. So, according to the explanation above, I should sum all the contributions for each "appearance" and use that as the update of each value of the kernel.

I tried to implement this but I eventually end up with all NaNs all over the network, so something is diverging. I believe the problem is in this line:

kernel{1} = kernel{1} + sum(sum(0.01*eta * delta_conv{1} .* x_out_conv{1}));
kernel{2} = kernel{2} + sum(sum(0.01*eta * delta_conv{2} .* x_out_conv{2}));

where kernel is a cell-array containing the two kernels (one for each convolution layer), eta is the learning rate, delta_conv is the gradient coming from the output of that layer and x_out_conv is the current output for each neuron in the layer.

That is constantly adding values, and it eventually goes out of control. Is that the way to update the weights of the kernels in a CNN?

EDIT:

To clarify, I'm using the convention from Hertz, Krogh & Palmer's Introduction to the Theory pf Neural Computation. When I talk about 'deltas', I'm referring to:

where $w$ are the weights, $E$ is the error function, $\eta$ the learning rate, $\zeta$ the desired outputs, $O$ the actual output, $g()$ the activation function, $\xi$ the inputs of the network and $h$ the linear output such that $g(h) = V$, the output of each neuron.

The rule to update the weights then becomes:

Answer 1

One of two things is probably going wrong. Either your learning rate is too high, or your gradient computation is incorrect. (Or both)

If it is your learning rate, simply reduce it by a few orders of magnitude and that should fix it.

In order to establish whether your gradient computations are correct or not, use the method of finite differences:

Pick one image or input, and start all the weights of your network at some deterministic value (0 perhaps). Then compute the loss, which is a single scalar value. Then pick one of your kernels, and increment one of the entries by a small epsilon (1E-6) perhaps. Compute the loss again. The derivative of the loss with respect to the weight is simply $\frac{\delta loss}{\epsilon}$. Now repeat this for all the weights in that one kernel. Check that it is equal to delta_conv.

Another possible problem might be weight initialization. Starting all the weights at 0, although not ideal, may help you determine if initialization is the culprit.

Finally, I recommend only using a single layer network while debugging, if you are not doing that already. Once that is working, you can add in more layers of your network and see if keeps working.

Answer 2

Concerning NaNs, you should also watch out for exploding gradients by simply clipping the computed gradient values to a maximum / minimum threshold.