How are feature maps connected between two layers in Theano/Caffe/TensorFlow?
For instance, if we have 32 feature maps in Conv Layer 1, and 64 feature maps in Conv Layer 2, with 64 kernels, how does the implementation connect the two layers? Is it fully connected? And if so, does it do an average across all inputs?
The second layer basically applies a 3D convolution instead of a 2D convolution. But, the 3D convolution kernel size in the last dimensions, which is used to represent color channels or different features, is chosen equal to the number of colors/features. Therefore, the result of the 3D convolution is still a 2D feature map for each 3D kernel.
For a black and white image the 2D convolutional layer actual applies a mathematical 2D convolution. So if you have your 100x100 image $I$ and a 3x3 kernel $k$, then, ignoring boundary conditions and assuming a stride of one, the feature map $f$ would be calculated as: $f_{i,j} = \sum\limits_{m=-1}^1 \sum\limits_{n=-1}^1 k_{m,n}, I_{i+m,j+n}$.
In your case, you have 32 feature maps $f_{1, \ldots, 32}$. This would result in an output shaped $(98,98,32)$, this is three dimensional! 2 pixels per dimensions have been lost because of the kernel size. Now, your very valid question is how a supposedly 2D-convolution is applied to a 3D shape.
One wrong assumption could be, that each feature map per sé undergoes a 2D convolution with 3x3 sized kernels. If the 2nd layer has 64 kernels, then this would result in $32*64$ results. But, in this case different features can't intermingle! If one feature is a nose and another a mouth, then subsequent convolutional layers could not reason that a certain area contains both, the mouth and the nose, only the fully connected layer at the end would be able to do this.
Instead, the $(98,98,32)$ output undergoes a 3D convolution with a 3x3x32 sized kernel in order to produce one feature map. That 2nd layer feature map is calculated as: $f_{i,j} = \sum\limits_{m=-1}^1 \sum\limits_{n=-1}^1 \sum\limits_{o=1}^{32} k_{m,n,o}, I_{i+m,j+n,o}$.
This is the same procedure which would have been done for a colored input image as can be seen in this animation of a pseudo-2D convolution with 3D-inputs by Martin Görner from here:
For colored input (as opposed to different feature maps), it makes even more sense that the convolution kernels have to be 3D convoluted instead of undergoing separate 2D convolutions. Else, color changes, e.g., from green to red, would be undetectable by the convolutional layer. Below are some example learned convolution kernels from "ImageNet Classification with Deep Convolutional Neural Networks" by Krizhevsky, Sutskever, and Hinton from 2012
As can be seen, the kernels are colored images, therefore, they are 3D kernels.
Typically one places a sub-sampling layer between two convolutional layers. The most commonly used sub-sampling layer in computer vision is max pooling.
Check out this tutorial for an example in Theano: Convolutional Neural Networks (LeNet)
