There are couple of reasons padding is important:

 1. It's easier to design networks if we preserve the `height` and `width` and don't have to worry too much about tensor dimensions when going from one layer to another because dimensions **will just "work"**.

 2. It allows us to design **deeper networks**. Without padding, reduction in volume size would reduce too quickly.

 3. Padding actually **improves performance by keeping information at the borders**.

Quote from Stanford lectures: *"In addition to the aforementioned benefit of keeping the spatial sizes constant after CONV, doing this actually improves performance. If the CONV layers were to not zero-pad the inputs and only perform valid convolutions, then the size of the volumes would reduce by a small amount after each CONV, and the information at the borders would be “washed away” too quickly."* - [source][1]

 4. As @dontloo already said, new network architectures need to concatenate convolutional layers with 1x1, 3x3 and 5x5 filters and it wouldn't be possible if they didn't use padding because dimensions wouldn't match. Check this image of inception module to understand better why padding is useful here.

[![enter image description here][2]][2]


  [1]: http://cs231n.github.io/convolutional-networks/
  [2]: https://i.sstatic.net/ldTdM.png