According to this guide, when applying a kernel to an input volume, the kernel always has to have the same depth as the input volume. When using images, I figured, that means the color channels.
Example 1. For example, suppose that the input volume has size [32x32x3], (e.g. an RGB CIFAR-10 image). If the receptive field (or the filter size) is 5x5, then each neuron in the Conv Layer will have weights to a [5x5x3] region in the input volume, for a total of 5*5*3 = 75 weights (and +1 bias parameter). Notice that the extent of the connectivity along the depth axis must be 3, since this is the depth of the input volume.
Example 2. Suppose an input volume had size [16x16x20]. Then using an example receptive field size of 3x3, every neuron in the Conv Layer would now have a total of 3*3*20 = 180 connections to the input volume. Notice that, again, the connectivity is local in space (e.g. 3x3), but full along the input depth (20).
Since we are calculating a dot product for every neuron in the following layer, that means that every feature map is 2 dimensional again with no color channel, right? So that means, the kernels for all succeeding layers have to be 2-dimensional again?
Also, if anyone knows working code in C++ that is not using any library like TensorFlow, I would much appreciate that. That would really help me understand all this way better than the rather abstract documents about the topic.