I am trying to build a Convolutional Neural Network (CNN) from scratch to learn its basics. However, I haven't found any information about how the weights from the kernels get updated in each iteration using backpropagation.

The only explanation I found on the internet was this one but I'm not sure if that is right or if I didn't implement it correctly in MATLAB.

I'm using "same" convolution and $3\times3\times1$ kernels. As far as I understand, all the positions of the kernels appear at all the locations of the images. So, according to the explanation above, I should sum all the contributions for each "appearance" and use that as the update of each value of the kernel.

I tried to implement this but I eventually end up with all NaNs all over the network, so something is diverging. I believe the problem is in this line:

kernel{1} = kernel{1} + sum(sum(0.01*eta * delta_conv{1} .* x_out_conv{1}));
kernel{2} = kernel{2} + sum(sum(0.01*eta * delta_conv{2} .* x_out_conv{2}));

where kernel is a cell-array containing the two kernels (one for each convolution layer), eta is the learning rate, delta_conv is the gradient coming from the output of that layer and x_out_conv is the current output for each neuron in the layer.

That is constantly adding values, and it eventually goes out of control. Is that the way to update the weights of the kernels in a CNN?


To clarify, I'm using the convention from Hertz, Krogh & Palmer's Introduction to the Theory pf Neural Computation. When I talk about 'deltas', I'm referring to:

enter image description here

where $w$ are the weights, $E$ is the error function, $\eta$ the learning rate, $\zeta$ the desired outputs, $O$ the actual output, $g()$ the activation function, $\xi$ the inputs of the network and $h$ the linear output such that $g(h) = V$, the output of each neuron.

The rule to update the weights then becomes:

enter image description here

  • $\begingroup$ Hey.. I want to ask did you find the solution? I am facing the exact same problem $\endgroup$
    – DuttaA
    Jun 20, 2018 at 10:50

2 Answers 2


One of two things is probably going wrong. Either your learning rate is too high, or your gradient computation is incorrect. (Or both)

If it is your learning rate, simply reduce it by a few orders of magnitude and that should fix it.

In order to establish whether your gradient computations are correct or not, use the method of finite differences:

Pick one image or input, and start all the weights of your network at some deterministic value (0 perhaps). Then compute the loss, which is a single scalar value. Then pick one of your kernels, and increment one of the entries by a small epsilon (1E-6) perhaps. Compute the loss again. The derivative of the loss with respect to the weight is simply $\frac{\delta loss}{\epsilon}$. Now repeat this for all the weights in that one kernel. Check that it is equal to delta_conv.

Another possible problem might be weight initialization. Starting all the weights at 0, although not ideal, may help you determine if initialization is the culprit.

Finally, I recommend only using a single layer network while debugging, if you are not doing that already. Once that is working, you can add in more layers of your network and see if keeps working.

  • $\begingroup$ Thanks for your answer. There is something I don't understand. delta_conv is $256\times256$ (the size of the output of that layer) while the kernels are $3\times3$. So what are you referring to when you say that I should check if "it is equal to data_conv" ? $\endgroup$
    – Tendero
    Jul 15, 2017 at 23:10
  • $\begingroup$ I'm not sure what you mean when you said delta_conv is the gradient coming from the output of the layer. What do you mean mathematically? At some point, you must have computed $\frac{\partial \text{loss}}{\partial \text{kernel_weights}}$. That's what you want to compare with the finite differences method. $\endgroup$
    – shimao
    Jul 15, 2017 at 23:12
  • $\begingroup$ I'm following the convention in Hertz, Krogh & Palmer's Introduction to the Theory of Neural Computation. If you don't have access to it, let me know so I can edit my question so that it is more understandable (I thought it was standard nomenclature, but what I named $\delta$ is not $\frac{\partial E}{\partial w_k}$). $\endgroup$
    – Tendero
    Jul 15, 2017 at 23:17
  • $\begingroup$ Yes, please do edit the question to specify what $\delta$ is $\endgroup$
    – shimao
    Jul 15, 2017 at 23:19
  • $\begingroup$ Done, I've added some definitions. $\endgroup$
    – Tendero
    Jul 15, 2017 at 23:54

Concerning NaNs, you should also watch out for exploding gradients by simply clipping the computed gradient values to a maximum / minimum threshold.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.