There are already some very good answers here. I want to add some more details about the image border effects (which were already mentioned) which depend on the padding type used.
There are 3 relevant padding types in deep learning:
- valid (no padding at all)
- same (keep image size by adding zeros around the image - that's what you are talking about and that's what most of the time is called "zero padding" in deep learning context)
- full (ensure all pixels have same influence on output, even more zeros are added around image, the output is larger than the input)
Here is a sketch how these 3 padding types work, with x the size 3 input, k the size 3 kernel (which is shifted to all possible locations), y the output and 0 indicates zero padding:
valid:
xxx
kkk
y
same:
0xxx0
kkk
kkk
kkk
yyy
full:
00xxx00
kkk
kkk
kkk
kkk
kkk
yyyyy
Let's look at how much influence (how often the kernel "touches" the pixel) a pixel in a 10x10 input image that is processed by a 3x3 convolution kernel has on the output (left same, right valid padding):
As you can see, with same padding the border pixels have less influence than the central pixels, so it is not true that same padding removes boundary effects completely (as one can sometimes read on the internet). For valid padding, this problem is even more severe. With full padding, on the other hand, all pixels have the same influence on the output.
As the network gets deeper, the problem gets more intense - both for valid and same padding.
I summarized my finding on the padding experiments I did, and here is an interesting paper about this topic.