Here is something that bothers me about Convolutional Neural Networks, hopefully I can explain this well:
While the input to a given layer has channels, and the output FROM that layer has channels, the output for a given filter in that layer lacks channels. It is just a simple width*height image.
In a sense, we can say that - so far as that filter is concerned - the input channels have been compressed into something simpler.
But as a result, for any given number in the filter output, we can't tell if the number has a high value because of what the filter saw in (for example) an input channel representing vertical edges, or because of what the filter saw in an input channel representing horizontal edges, etc.
So it seems like we lose the "type" of information that was gained by the previous layer. Why isn't this a problem?