Convolutional Layers: To pad or not to pad?

Question

AlexNet architecture uses zero-paddings as shown in the pic. However, there is no explanation in the paper why this padding is introduced.

Standford CS 231n course teaches we use padding to preserve the spatial size:

I am curious if that is the only reason for zero padding? Can anyone explain the rationale behind zero padding? Thanks!

Reason I am asking

Let's say I don't need to preserve the spatial size. Can I just remove padding then w/o loss of performance? I know it results in very fast decrease in spatial size as we go to deeper layers, but I can trade-off that by removing pooling layers as well.

Answer 1

There are couple of reasons padding is important:

1. It's easier to design networks if we preserve the height and width and don't have to worry too much about tensor dimensions when going from one layer to another because dimensions will just "work".

2. It allows us to design deeper networks. Without padding, reduction in volume size would reduce too quickly.

3. Padding actually improves performance by keeping information at the borders.

Quote from Stanford lectures: "In addition to the aforementioned benefit of keeping the spatial sizes constant after CONV, doing this actually improves performance. If the CONV layers were to not zero-pad the inputs and only perform valid convolutions, then the size of the volumes would reduce by a small amount after each CONV, and the information at the borders would be “washed away” too quickly." - source

4. As @dontloo already said, new network architectures need to concatenate convolutional layers with 1x1, 3x3 and 5x5 filters and it wouldn't be possible if they didn't use padding because dimensions wouldn't match. Check this image of inception module to understand better why padding is useful here.

Answer 2

It seems to me the most important reason is to preserve the spatial size. As you said, we can trade-off the decrease in spatial size by removing pooling layers. However many recent network structures (like residual nets, inception nets, fractal nets) operate on the outputs of different layers, which requires a consistent spatial size between them.

Another thing is, if no padding, the pixels in the corner of the input only affect the pixels in the corresponding corner of the output, while the pixels in the centre contribute to a neighbourhood in the output. When several no-padding layers get stacked together, the network sort of ignores the boarder pixels of the image.

Just some of my understandings, I believe there are other good reasons.

Answer 3

There are already some very good answers here. I want to add some more details about the image border effects (which were already mentioned) which depend on the padding type used.

There are 3 relevant padding types in deep learning:

• valid (no padding at all)
• same (keep image size by adding zeros around the image - that's what you are talking about and that's what most of the time is called "zero padding" in deep learning context)
• full (ensure all pixels have same influence on output, even more zeros are added around image, the output is larger than the input)

Here is a sketch how these 3 padding types work, with x the size 3 input, k the size 3 kernel (which is shifted to all possible locations), y the output and 0 indicates zero padding:

valid:
xxx
kkk
 y

same:  
0xxx0
kkk
 kkk
  kkk
 yyy

full:
00xxx00
kkk
 kkk
  kkk
   kkk
    kkk
 yyyyy

Let's look at how much influence (how often the kernel "touches" the pixel) a pixel in a 10x10 input image that is processed by a 3x3 convolution kernel has on the output (left same, right valid padding):

As you can see, with same padding the border pixels have less influence than the central pixels, so it is not true that same padding removes boundary effects completely (as one can sometimes read on the internet). For valid padding, this problem is even more severe. With full padding, on the other hand, all pixels have the same influence on the output.

As the network gets deeper, the problem gets more intense - both for valid and same padding.

I summarized my finding on the padding experiments I did, and here is an interesting paper about this topic.

Answer 4

Great question. Drag0 explained nicely but I agree, something is amiss.

It's like looking at a photograph and having to deal with the border. In real life, you can move your eyes to look further; No real borders exist. So it is a limitation of the medium.

Besides preserving size, does it matter? I am not aware of a satisfactory answer but I conjecture (unproven) that with experiments on attention and occlusion (partial objects), we don't need the information lost on the borders. If you were to do something smarter (say copy the pixel next to it), it wouldn't change the answer though I have not experimented myself. Padding with 0s is fast and preserves size, so that's why we do it.

Answer 5

Elaborating on keeping information at the border, basically, the pixel at the corner (green shaded) when done convolution upon would just be used once whereas the one in the middle, like shaded red, would contribute to the resulting feature map multiple times.Thus, we pad the image See figure: 2.

Answer 6

this is my thinking. zero padding is important at initial time for keeping the size of ouput feature vector. and its someone above said that zero padding has more performance.

but how about in last time? image feature vector resolution is very small, and pixel value means a kind of vector of some global size.

I think in last case some kind of mirroring is better then zero padding.

Answer 7

I'll try to tell from the regard of information that when is it okay to pad and when it is not.

Let's for base case take the example of tensorflow padding functionality. It provides two scenarios, either "Valid" or "same". Same will preserve the size of the output and will keep it the same as that of the input by adding suitable padding, while valid won't do that and some people claim that it'll lead to loss of information, but, here's the catch.

This information loss depends on the size of the kernel or the filter you're using. For example, let's say you have a 28x28 image and the filter size is 15x15(let's say). The output should have dimension 16x16, but if you pad using "same" in tensorflow it will be 28x28. Now the 12 rows and 12 columns in themselves don't carry any meaningful information but are still there as a form of noise. And we all know how much susceptible deep learning models are towards the noise. This can degrade the training a lot. So if you're using big filters, better not go with padding.