Why do people use Zero-Padding in Convolutional Neural Networks? I am wondering why people usually pad with zeros instead of e.g., using the min-value. Zero-padding, in my opinion, makes sense if you have input images with a pixel range [0, 255] or [0, 1] (after normalization). However, for hidden layer representations, unless you use e.g., ReLU or Logistic Sigmoid activation functions, it doesn't make quite sense to me.
E.g., if you have normalized your input images in range [-0.5, 0.5] as it is commonly done, then using Zero padding does not make sense to me (as opposed to padding with -0.5). Same goes for tanH activations, as the gradient is the steepest at 0.
So, I am wondering why people use Zero-padding everywhere nonetheless?
 A: Zero-padding is a generic way to (1) control the shrinkage of dimension after applying filters larger than 1x1, and (2) avoid loosing information at the boundaries, e.g. when weights in a filter drop rapidly away from its center. 
For a specific input, activation function, or loss function, a variant might perform better, i.e. utilizing domain knowledge. However, the key for zero-padding is "being generic". For example, a completely different padding would be "reflection padding" that, instead of a specific value, puts a mirror of input outside the boundaries. We could try reflection padding and if it gives better results, then we might look for a justification based on the task, activation function, etc.
As an example related to comments, assume black and white images with $\text{tanh}$ activation functions (between $-1$ and $1$), we may opt for $(-1)$-padding instead of $0$-padding. If we reverse the black and white in the image, now $1$-padding would be more justified for the same reason. 
A: If you consider the central limit theorem, input data will follow a normal distribution with a constant mean. Thus, if the input data are normalized, the mean will be close to 0. So padding with 0 (the mean) doesn't affect the distribution. I have done some testings in my research, which show the output of batch normalization will follow normal distributions with the mean close to 0. Also if you know the convolution operation in traditional signal processing, you can find that zero padding is just the standardized way.
