There are already some very good answers here. I want to add some more details about the image border effects (which were already mentioned) that are connected to the type of padding used.

There are 3 relevant padding types in deep learning:

• valid (no padding)
• same (keep image size by adding zeros around the image - that's what you are talking about and that's what most of the time is called "zero padding" in deep learning context)
• full (ensures all pixels have same influence on output, even more zeros are added around image, the output is larger than the input)

Here a sketch how these 3 padding types work, with x the size 3 input, k the size 3 kernel (which is shifted in all possible positions), y the output and 0 indicates zero padding:

valid:
xxx
kkk
 y

same:  
0xxx0
kkk
 kkk
  kkk
 yyy

full:
00xxx00
kkk
 kkk
  kkk
   kkk
    kkk
 yyyyy

Let's look at how much influence a pixel in a 10x10 input image that is processed by a 3x3 convolution kernel has on the output (left same, right valid padding):

As you can see, with same padding the border pixels have less influence than the central pixels, so it is not true that same padding removes boundary effects completely (as one can sometimes read in the internet). For valid padding, this problem is more severe. With full padding, on the other hand, all pixels have the same influence on the output.

When the network gets deeper, this problem gets more severe - for both valid and same padding.

I wrote an article about the padding experiments I did, and here is an interesting paper about this topic.

There are already some very good answers here. I want to add some more details about the image border effects (which were already mentioned) which depend on the padding type used.

There are 3 relevant padding types in deep learning:

• valid (no padding at all)
• same (keep image size by adding zeros around the image - that's what you are talking about and that's what most of the time is called "zero padding" in deep learning context)
• full (ensure all pixels have same influence on output, even more zeros are added around image, the output is larger than the input)

Here is a sketch how these 3 padding types work, with x the size 3 input, k the size 3 kernel (which is shifted to all possible locations), y the output and 0 indicates zero padding:

valid:
xxx
kkk
 y

same:  
0xxx0
kkk
 kkk
  kkk
 yyy

full:
00xxx00
kkk
 kkk
  kkk
   kkk
    kkk
 yyyyy

Let's look at how much influence (how often the kernel "touches" the pixel) a pixel in a 10x10 input image that is processed by a 3x3 convolution kernel has on the output (left same, right valid padding):

As you can see, with same padding the border pixels have less influence than the central pixels, so it is not true that same padding removes boundary effects completely (as one can sometimes read on the internet). For valid padding, this problem is even more severe. With full padding, on the other hand, all pixels have the same influence on the output.

As the network gets deeper, the problem gets more intense - both for valid and same padding.

I summarized my finding on the padding experiments I did, and here is an interesting paper about this topic.