About depth in convolution layers

Question

I am trying to build a CNN from scratch to understand how it works. I am having problems designing the convolution layers, specifically with its depth.

Suppose that each image I put at the input is in grayscale and of size 256x256. That means that the initial dimension is 256x256x1. According to what I've read and understood so far, at each convolution layer one can exctract several feature maps (i.e. convolve the image with a different number of kernels). That number equals the depth.

Therefore, if I decide to use 3 kernels per layer, and assuming I zero-pad the image so that the convolution yields an image of the same size and that the stride equals 1:

• At the output of the first convolution layer, I'll have an output of 256x256x3.
• After 2x2 max pooling, I'll have 128x128x3.
• That pattern enters the next convolution layer, which will have 3 kernels as well, thus the output will be 128x128x9.

My questions are:

1. Is the last description okay? If I use three kernels per layer, will the third dimension at the output of the $N$-th layer be equal to $3^N$?
2. Can I represent these 'cubic' matrices (i.e. with third dimension greater than 1) as vectors at the input of the fully-connected network? For instance, if I have a matrix of dimensions 8x8x27 at the end of the convolution stages, can I order those numbers as a vector of dimension 1x1728 ($8\cdot8\cdot27=1728$) so that the fully-connected network is easier to implement?

Answer 1

When you convolve a layer of size 128x128x3 by 3 kernels, you actually end up with a 128x128x3 tensor. This is because each kernel is a 3x3x3 tensor, so you don't end up with exponential growth you described. Each kernel compresses all 3 of the "feature maps" in the last layer into one value. If you only had one kernel, a 128x128x3 input would result in a 128x128x1 output.

Mathematically, if single channel convolution was

$$Y_{i,j} = \sum_{\delta_i, \delta_j} X_{i+\delta_i, j+\delta_j} \cdot K_{\delta_i, \delta_j} $$

Then with multiple channels and one kernel we have

$$Y_{i,j} = \sum_{\delta_i, \delta_j,k} X_{i+\delta_i, j+\delta_j,k} \cdot K_{\delta_i, \delta_j,k} $$

Second, yes, flattening the tensor at the end of the convolutional layers into a vector which is input into fully connected layers is standard practice. Alternatively, you can also average across each feature map, so that a 8x8x1024 tensor is transformed into a 1024 dimension vector.