11
$\begingroup$

I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared weights, so as to save time during training. But, how does one adapt backpropagation so it can be used in a convolution neural network. In backpropagation, one uses a formula similar to this to train the weights.

New Weight  = Old Weight +  LEARNING_RATE * 1 * Output Of InputNeuron * Delta

However, since in convolutional neural networks, the weights are shared, each weight is used with multiple neurons, so how do I decide which Output of InputNeuron is used?

In other words, since the weights are shared, how do I decide how much to change the weights by?

$\endgroup$
13
$\begingroup$

You need to first calculate all your updates as if the wieghts weren't shared, but just store them, don't actually do any updating yet.

Let $w_k$ be some weight that appears at locations $I_k = \{(i,j) \colon w_{i,j} = w_k\}$ in your network and $\Delta w_{i,j} = -\eta \frac{\partial J}{\partial w_{i,j}} $ where $\eta$ is the learning rate and $J$ is your objective function. Note that at this point if you didn't have weight sharing you would just upade $w_{i,j}$ as $$ w_{i,j} = w_{i,j} + \Delta w_{i,j}. $$ To deal with the shared weights you need to sum up all the individual updates. So set $$ \Delta w_k = \sum_{(i,j) \in I_k} \Delta w_{i,j} $$ and then update $$ w_k = w_k + \Delta w_k. $$

$\endgroup$
  • 1
    $\begingroup$ Do you have any references for this? For further reading $\endgroup$ – Jon Aug 15 '18 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.