Let's say I have RGB input data (3 channels) and a convolutional layer which has just one filter with a depth of 3. The output data will have a depth of 1 if we build the sum over the results of the convolution of every channel. But why do sum up the results? Why not build the average or add 17 all time?
It seems like we might lose information due to the summation. For example, if there is a positive edge on the red channel but a negative edge on the blue channel they will cancel each other out. Okay, the weights can be different for each channel that might help, but I still don't see the advantage of a summation over other operations.
R (1. channel) conv Filter 1 [x:x:1] \ \ G (2. channel) conv Filter 1 [x:x:2] => Sum => output [x:x:1] WHY? / B (3. channel) conv Filter 1 [x:x:3] /
Here is a much better graphic (scroll down to the gif). http://cs231n.github.io/convolutional-networks/#conv