13
$\begingroup$

I am wondering why people usually pad with zeros instead of e.g., using the min-value. Zero-padding, in my opinion, makes sense if you have input images with a pixel range [0, 255] or [0, 1] (after normalization). However, for hidden layer representations, unless you use e.g., ReLU or Logistic Sigmoid activation functions, it doesn't make quite sense to me.

E.g., if you have normalized your input images in range [-0.5, 0.5] as it is commonly done, then using Zero padding does not make sense to me (as opposed to padding with -0.5). Same goes for tanH activations, as the gradient is the steepest at 0.

So, I am wondering why people use Zero-padding everywhere nonetheless?

Improve this question
$\endgroup$
1
  • $\begingroup$ Zero padding (i.e. "padding with zeros"), helps CNNs encode position information of content features it has identified, by providing an obvious reference point. refs (2021): arxiv.org/pdf/2101.12322.pdf $\endgroup$
    – russian_spy
    Commented Apr 18, 2022 at 16:03

2 Answers 2

Reset to default
6
$\begingroup$

Zero-padding is a generic way to (1) control the shrinkage of dimension after applying filters larger than 1x1, and (2) avoid loosing information at the boundaries, e.g. when weights in a filter drop rapidly away from its center.

For a specific input, activation function, or loss function, a variant might perform better, i.e. utilizing domain knowledge. However, the key for zero-padding is "being generic". For example, a completely different padding would be "reflection padding" that, instead of a specific value, puts a mirror of input outside the boundaries. We could try reflection padding and if it gives better results, then we might look for a justification based on the task, activation function, etc.

As an example related to comments, assume black and white images with $\text{tanh}$ activation functions (between $-1$ and $1$), we may opt for $(-1)$-padding instead of $0$-padding. If we reverse the black and white in the image, now $1$-padding would be more justified for the same reason.

Improve this answer
$\endgroup$
6
  • $\begingroup$ > "For a specific input, activation function, or loss function, a variant might perform better," That makes sense, but I am wondering if choosing always the min value of the activation function would already be an improvement over using 0. Same for images after normalization. $\endgroup$
    – resnet
    Commented Apr 1, 2019 at 14:03
  • $\begingroup$ Because like you said, one of the goals is to avoid shrinking and also information loss at the edges. I usually think of the min value as "placeholder pixels (e.g., black in b/w images)" E.g., if MNIST was b/w and normalized to a [-0.5, 0.5] range, zero padding would add gray pixels, which is not immediately intuitive $\endgroup$
    – resnet
    Commented Apr 1, 2019 at 14:09
  • $\begingroup$ Thanks for the follow up via your edit. Regarding "Another important note is that the filters are doing weighted sum over input patches, thus we generally want a padding that is as neutral as possible for the summation, i.e. $f(i,j)= w(i,j) \times x(i, j) +w(i+1, j+1) \times 0 +..$. which further justifies the zero-padding as a generic solution." --> I am not sure if this is as neutral as possible, because when using e.g., tanH, you have a maximum gradient when doing back propagation when the activations are 0. For ReLU, this reasoning may make sense though. $\endgroup$
    – resnet
    Commented Apr 1, 2019 at 17:29
  • 1
    $\begingroup$ @resnet you are right, I was about to say because 0 is equal to "non-existent" input, but again we can define non-existent as "-1"! So I removed that part. $\endgroup$
    – Esmailian
    Commented Apr 1, 2019 at 17:47
  • $\begingroup$ Makes sense :). I think in most cases, something like "Min-Padding" would make sense as a more general recommendation instead of Zero-Padding? I.e., always choosing the smallest activation value. Currently can't think of a counter example as activation functions I know are usually monotonic. What do you think? $\endgroup$
    – resnet
    Commented Apr 1, 2019 at 18:01
2
$\begingroup$

If you consider the central limit theorem, input data will follow a normal distribution with a constant mean. Thus, if the input data are normalized, the mean will be close to 0. So padding with 0 (the mean) doesn't affect the distribution. I have done some testings in my research, which show the output of batch normalization will follow normal distributions with the mean close to 0. Also if you know the convolution operation in traditional signal processing, you can find that zero padding is just the standardized way.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.