How are kernels applied to feature maps to produce other feature maps? I am trying to understand the convolution part of convolutional neural networks. Looking at the following figure:

I have no problems understanding the first convolution layer where we have 4 different kernels (of size $k \times k$), which we convolve with the input image to obtain 4 feature maps.
What I do not understand is the next convolution layer, where we go from 4 feature maps to 6 feature maps. I assume we have 6 kernels in this layer (consequently giving 6 output feature maps), but how do these kernels work on the 4 feature maps shown in C1? Are the kernels 3-dimensional, or are they 2-dimensional and replicated across the 4 input feature maps?
 A: There is not a one-to-one correspondence between layers and kernels necessarily. That depends on the particular architecture. The figure you posted suggests that in the S2 layers you have 6 feature maps, each combining all feature maps of the previous layers, i.e. different possible combinations of the features.
Without more references I cannot say much more. See for example this paper
A: The kernels are 3-dimensional, where width and height can be chosen, while the depth is equal to the number of maps in the input layer - in general. 
They are certainly not 2-dimensional and replicated across the input feature maps at the same 2D location! That would mean a kernel wouldn't be able to distinguish between its input features at a given location, since it would use one and the same weight across the input feature maps!
A: Table 1 and Section 2a of Yann LeCun's "Gradient Based Learning Applied to Document Recognition" explains this well: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf  Not all regions of the 5x5 convolution are used to generate the 2nd convolutional layer.
A: This article can be helpful: Understanding Convolution in Deep Learning by Tim Dettmers from March 26
It doesn't really answers the question because it explains only the first convolution layer, but contains good explanation of basic intuition about convolution in CNNs. It also describes deeper mathematical definition of convolution. I think it is related to question topic.
