Let's say I have RGB input data (3 channels) and a convolutional layer which has just one filter with a depth of 3. The output data will have a depth of 1 if we build the sum over the results of the convolution of every channel. But why do sum up the results? Why not build the average or add 17 all time?

Some thoughts:
It seems like we might lose information due to the summation. For example, if there is a positive edge on the red channel but a negative edge on the blue channel they will cancel each other out. Okay, the weights can be different for each channel that might help, but I still don't see the advantage of a summation over other operations.

R (1. channel) conv Filter 1 [x:x:1] \
G (2. channel) conv Filter 1 [x:x:2]    => Sum => output [x:x:1] WHY?
B (3. channel) conv Filter 1 [x:x:3] /

Here is a much better graphic (scroll down to the gif). http://cs231n.github.io/convolutional-networks/#conv

  • $\begingroup$ Your proposed architecture is definitely not the standard one. So the question 'Why?' sounds odd $\endgroup$
    – Maxim
    Feb 3, 2018 at 20:16
  • $\begingroup$ Perhaps I have phrased it poorly. But I think this is actually the standard. You convolute your input with three channels with a filter with three channels and then sum up the results so that you get an output which has only one channel/depth. When you use more than one filter in your conv layer then the output will get additional channels for each added filter. $\endgroup$
    – siva
    Feb 3, 2018 at 23:30
  • $\begingroup$ ok, I got it. You're talking about input channels. See my answer $\endgroup$
    – Maxim
    Feb 4, 2018 at 0:10
  • $\begingroup$ This exact question came up with a colleague the other day. Useful! $\endgroup$
    – vpipkt
    Jan 21, 2019 at 14:58
  • $\begingroup$ READERS NOTE: The architecture is indeed "standard". In this case, we are talking about the input layer. $\endgroup$ May 15, 2020 at 8:31

1 Answer 1


Okay, the weights can be different for each channel that might help, but I still don't see the advantage of a summation over other operations.

Exactly. You miss the fact that weights are learnable. Of course, initially it's possible that the edges from different channels will cancel each other and the output tensor will lose this information. But this will result in big loss value, i.e., big backpropagation gradients, that will tweak the weights accordingly. In reality, the network learns to capture the edges (or corners or more complex patterns) in any channel. When the filter does not match the patch, the convolution result is very close to zero, rather than large negative number, so that nothing is lost in the sum. (In fact, after some training, most of the values in the kernels are close to zero.)

The reason to do a summation is because it's efficient (both forward and backward operations are vectorizable) and allows nice gradients flow. You can view this as a sophisticated linear layer with shared weights. If the concern you express was actual, you would see the same problem in all linear layers in any network: when different features are summed up with some weights, they can cancel each other out, right? Luckily, this does not happen (unless the features are correlated, e.g., specially crafted), due to reasons described earlier, so the linear operation is the crucial building block of any neural network.

  • 1
    $\begingroup$ Thanks for the great answer. Especially comparing it to a sophisticated linear layer convinced me that it works. But I am still not quite sold on why summation is the best option. I get that it's easy to vectorize. Do you know a paper which compares different operations? $\endgroup$
    – siva
    Feb 4, 2018 at 1:11
  • $\begingroup$ @siva Taking the mean average is equivalent to dividing the filter weights by N and then summing as usual. Do you have another operation in mind which is (a) computationally efficient, (b) differentiable, and (c) doesn't suffer from "vanishing gradient"-like issues? $\endgroup$ May 15, 2020 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.