5
$\begingroup$

My question is regarding the answer to this question: Training a convolution neural network

It seems like the answer is saying to change all the weights in a given filter by the same amount in the same direction?

Won't the CNN (convolutional neural network) be unable to learn appropriate filters this way?

See the diagrams representing the weights here: http://jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/

The 2x2 delta is backpropagated onto a 2x2 filter which creates a 3x3 grid of the change in weights If you are supposed to add up those 9 values and then add them to the original 2x2 filter at all positions it will just change each value in the 2x2 filter by the same amount and in the same directions. Is that the correct way to update weights in a CNN? Or are you supposed to update each weight (each value in the 2x2 filter) individually by its own gradient like in a normal artificial neural network ANN?

$\endgroup$
  • 1
    $\begingroup$ could you briefly summarize the answer you are asking about, so as to make your question self-contained? (if the answer you are referring to is deleted in the future, for example). Thanks $\endgroup$ – Antoine Jun 16 '17 at 15:20
2
$\begingroup$

A convolutional network layer is a just a fully-connected layer where two things are true:

  • certain connections are removed; means their weights are forced to be constant zero. So you can just ignore these connections/weights
  • each of the non-zero weights is shared across multiple connections, ie across multiple pairs of input/output neurons

When a weight is shared across connections, the gradient update for that weight is the sum of the gradients across all the connections that share the same weight.

So then looking at your questions:

Or are you supposed to update each weight (each value in the 2x2 filter) individually by its own gradient like in a normal artificial neural network ANN?

Each weight is updated just like in a 'normal ann', by which case I interpret this to mean in a fully-connected, or 'dense', or 'linear' layer, right? However, as stated, the update applied to each of the weights will be the sum of the gradients from all the connections that share that weight.

The 2x2 delta is backpropagated onto a 2x2 filter which creates a 3x3 grid of the change in weights If you are supposed to add up those 9 values and then add them to the original 2x2 filter at all positions it will just change each value in the 2x2 filter by the same amount and in the same directions. Is that the correct way to update weights in a CNN?

The gradients 'flow back' to where they came. In the forward pass, each weight in the CNN 'kernel' will be used to calculate the output of multiple connections. In the backward pass, the gradients will 'flow backwards', along those exact same connections, onto the originating weights.

In the worst case, rather than thinking of the CNN as some magical thing, you can, as stated, consider a CNN to be a standard fully-connected/linear layer, with many connections/weights forced to be zero, and the remaining weights shared across multiple connections. Then, you can use your existing knowledge for fully-connected / linear layers, to handle the CNN layer.

For background, some conceptual aims/motivations of a CNN layer compared to a fully-connected layer are:

  • enforces a prior on adjacency. Since pixels in images tend to have more correlation and relationships with adjacent pixels, so we keep only connections between adjacent input/output pixels, and remove the other connections
  • partly to remove parameters, to avoid over-fitting, and partly because we want to enforce a prior on invariance to translation, we share the weights such that input pixels in one region will give identical outputs, no matter where in the image they are, but simply translated in the output image
$\endgroup$
1
$\begingroup$

In the answer you reference, the goal is to calculate $w_k$, one of the weights in one filter in a (convolutional) neural network. The update formulas for any parameter in a neural network, using gradient descent, is quite simple if you just express it as a derivative of the cost function $J$:

$\Delta w_k = -\eta \frac{\partial J}{\partial w_k}$

But, computing $\frac{\partial J}{\partial w_k}$ is where things get tricky. In a convolutional neural network, this weight is shared, so you need to calculate

$\frac{\partial J}{\partial w_k} = \sum \frac{\partial J}{\partial w_l}$

where the sum is taken over all the occurrences of that weight. So, if you have a 2x2 filter and an 8x8 image (stride 1), the filter is applied 7x7=49 times and in each application of this filter the top left weight is used. So, the sum is over these 49 values. For the top right weight, however, the sum is still over 49 values, but it's now a different set of values and so the update is different.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.