I started coding backpropagation for a simple convnet and had some troubles understanding the algorithm. I do get the idea of weight update based on gradients, but because the filter kernel parameters are shared across the field, I am not sure hot to jointly process all gradients that should contribute to the update. How do I update the kernel values during backprop in this case?
Gradients flow back to where they came from. So any weight that contributed to an output gets updated with gradients from that output. In the case of a conv layer, a weight that contributed to multiple outputs get updated multiple times, once for each output. The updates are summed.
Updating the weights (kernels) through the convolutional layer is done by convolving the (l+1) delta with the (l+1) weight. This gives the delta for updating the kernel. For a detailed explanation, check out deepgrid. Also, check out the github repo on CNN here, that implements backpropagation and updating the weights on a simple CNN.