Convolutional neural network with images that have color channels

Question

According to this guide, when applying a kernel to an input volume, the kernel always has to have the same depth as the input volume. When using images, I figured, that means the color channels.

Example 1. For example, suppose that the input volume has size [32x32x3], (e.g. an RGB CIFAR-10 image). If the receptive field (or the filter size) is 5x5, then each neuron in the Conv Layer will have weights to a [5x5x3] region in the input volume, for a total of 5*5*3 = 75 weights (and +1 bias parameter). Notice that the extent of the connectivity along the depth axis must be 3, since this is the depth of the input volume.

Example 2. Suppose an input volume had size [16x16x20]. Then using an example receptive field size of 3x3, every neuron in the Conv Layer would now have a total of 3*3*20 = 180 connections to the input volume. Notice that, again, the connectivity is local in space (e.g. 3x3), but full along the input depth (20).

Since we are calculating a dot product for every neuron in the following layer, that means that every feature map is 2 dimensional again with no color channel, right? So that means, the kernels for all succeeding layers have to be 2-dimensional again?

Also, if anyone knows working code in C++ that is not using any library like TensorFlow, I would much appreciate that. That would really help me understand all this way better than the rather abstract documents about the topic.

Answer 1

Yes, when you start with a stack of two dimensional channels then each successive convolutional layer will use multiple stacks of two dimensional kernels. Those kernels are stacked to the same depth as the number of input channels, and each such stack sums up to produce a single output channel. So typically the kernel weights are stored in a four dimensional array $N_{kwidth} \times N_{kheight}\times N_{input channels} \times N_{output channels}$

The terms "channel" and "feature map" are interchangeable, a CNN treats a 3 colour image and a middle layer with 3 feature maps identically. Although in practice for most image processing tasks 3 feature maps would be quite low number to see in typical hidden layers.

In general after the input layer you don't think of the feature maps as being a meaningful image to a human, although you can plot them as images if you want, and the lower layers might be interpretable, looking similar to the input but with certain edges or textures enhanced.