Training a convolution neural network

Question

I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared weights, so as to save time during training. But, how does one adapt backpropagation so it can be used in a convolution neural network. In backpropagation, one uses a formula similar to this to train the weights.

New Weight  = Old Weight +  LEARNING_RATE * 1 * Output Of InputNeuron * Delta

However, since in convolutional neural networks, the weights are shared, each weight is used with multiple neurons, so how do I decide which Output of InputNeuron is used?

In other words, since the weights are shared, how do I decide how much to change the weights by?

Answer 1

You need to first calculate all your updates as if the wieghts weren't shared, but just store them, don't actually do any updating yet.

Let $w_k$ be some weight that appears at locations $I_k = \{(i,j) \colon w_{i,j} = w_k\}$ in your network and $\Delta w_{i,j} = -\eta \frac{\partial J}{\partial w_{i,j}} $ where $\eta$ is the learning rate and $J$ is your objective function. Note that at this point if you didn't have weight sharing you would just upade $w_{i,j}$ as $$ w_{i,j} = w_{i,j} + \Delta w_{i,j}. $$ To deal with the shared weights you need to sum up all the individual updates. So set $$ \Delta w_k = \sum_{(i,j) \in I_k} \Delta w_{i,j} $$ and then update $$ w_k = w_k + \Delta w_k. $$