I am wondering why people usually pad with zeros instead of e.g., using the min-value. Zero-padding, in my opinion, makes sense if you have input images with a pixel range [0, 255] or [0, 1] (after normalization). However, for hidden layer representations, unless you use e.g., ReLU or Logistic Sigmoid activation functions, it doesn't make quite sense to me.
E.g., if you have normalized your input images in range [-0.5, 0.5] as it is commonly done, then using Zero padding does not make sense to me (as opposed to padding with -0.5). Same goes for tanH activations, as the gradient is the steepest at 0.
So, I am wondering why people use Zero-padding everywhere nonetheless?