Why building the sum of a filter over several channels in a convolutional layer?

Let's say I have RGB input data (3 channels) and a convolutional layer which has just one filter with a depth of 3. The output data will have a depth of 1 if we build the sum over the results of the convolution of every channel. But why do sum up the results? Why not build the average or add 17 all time?

Some thoughts:
It seems like we might lose information due to the summation. For example, if there is a positive edge on the red channel but a negative edge on the blue channel they will cancel each other out. Okay, the weights can be different for each channel that might help, but I still don't see the advantage of a summation over other operations.

R (1. channel) conv Filter 1 [x:x:1] \
\
G (2. channel) conv Filter 1 [x:x:2]    => Sum => output [x:x:1] WHY?
/
B (3. channel) conv Filter 1 [x:x:3] /


EDIT:
Here is a much better graphic (scroll down to the gif). http://cs231n.github.io/convolutional-networks/#conv

• Your proposed architecture is definitely not the standard one. So the question 'Why?' sounds odd – Maxim Feb 3 '18 at 20:16
• Perhaps I have phrased it poorly. But I think this is actually the standard. You convolute your input with three channels with a filter with three channels and then sum up the results so that you get an output which has only one channel/depth. When you use more than one filter in your conv layer then the output will get additional channels for each added filter. – siva Feb 3 '18 at 23:30
• ok, I got it. You're talking about input channels. See my answer – Maxim Feb 4 '18 at 0:10
• This exact question came up with a colleague the other day. Useful! – vpipkt Jan 21 at 14:58