How does the DepthConcat operation in 'Going deeper with convolutions' work?

Question

Reading Going deeper with convolutions I came across a DepthConcat layer, a building block of the proposed inception modules, which combines the output of multiple tensors of varying size. The authors call this "Filter Concatenation". There seems to be an implementation for Torch, but I don't really understand, what it does. Can someone explain in simple words?

Answer 1

I don't think the output of the inception module are of different sizes.

For convolutional layers people often use padding to retain the spatial resolution.

The bottom-right pooling layer (blue frame) among other convolutional layers might seem awkward. However unlike conventional pooling-subsampling layers (red frame, stride>1), they used a stride of 1 in that pooling layer. Stride-1 pooling layers actually work in the same manner as convolutional layers, but with the convolution operation replaced by the max operation.

So the resolution after the pooling layer also stays unchanged, and we can concatenate the pooling and convolutional layers together in the "depth" dimension.

As shown in the above figure from the paper, the inception module actually keeps the spatial resolution.

Answer 2

I had the same question in mind as you reading that white paper and the resources you have referenced have helped me come up with an implementation.

In the Torch code you referenced, it says:

--[[ DepthConcat ]]--
-- Concatenates the output of Convolutions along the depth dimension
-- (nOutputFrame). This is used to implement the DepthConcat layer
-- of the Going deeper with convolutions paper :

The word "depth" in Deep learning is a little ambiguous. Fortunately this SO Answer provides some clarity:

In Deep Neural Networks the depth refers to how deep the network is but in this context, the depth is used for visual recognition and it translates to the 3rd dimension of an image.

In this case you have an image, and the size of this input is 32x32x3 which is (width, height, depth). The neural network should be able to learn based on this parameters as depth translates to the different channels of the training images.

So DepthConcat concatenates tensors along the depth dimension which is the last dimension of the tensor and in this case the 3rd dimension of a 3D tensor.

DepthConcat needs to make the tensors the same in all dimensions but the depth dimension, as the Torch code says:

-- The normal Concat Module can't be used since the spatial dimensions
-- of tensors to be concatenated may have different values. To deal with
-- this, we select the largest spatial dimensions and add zero-padding
-- around the smaller dimensions.

e.g.

A = tensor of size (14, 14, 2)
B = tensor of size (16, 16, 3)
result = DepthConcat([A, B])
where result with have a height of 16, a width of 16 and a depth of 5 (2 + 3). 

In the diagram above, we see a picture of the DepthConcat result tensor, where the white area is the zero padding, the red is the A tensor and the green is the B tensor.

Here's the pseudo code for DepthConcat in this example:

1. Look at tensor A and tensor B and find the biggest spatial dimensions, which in this case would be tensor B's 16 width and 16 height sizes. Since tensor A is too small and doesn't match the spatial dimensions of Tensor B's, it will need to be padded.
2. Pad the spatial dimensions of tensor A with zeros by adding zeros to the first and second dimensions making the size of tensor A (16, 16, 2).
3. Concatenate padded tensor A with tensor B along the depth (3rd) dimension.

I hope this helps somebody else who thinks the same question reading that white paper.