CNN filter sizes and padding

Question

I came across the following passage in http://cs231n.github.io/convolutional-networks/ with regards to filter sizes in CNNs. Purely because i have seen a number of networks with 5*5 conv filters without 2 padding - i wanted to check if this indeed is best practice. Any thoughts much appreciated.

The conv layers should be using small filters (e.g. $3\times 3$ or at most $5\times 5$), using a stride of $S=1$ , and crucially, padding the input volume with zeros in such way that the conv layer does not alter the spatial dimensions of the input. That is, when $F=3$ , then using $P=1$ will retain the original size of the input. When $F=5$ , $P=2$. For a general $F$, it can be seen that $P=(F−1)/2$ preserves the input size. If you must use bigger filter sizes (such as $7\times7$ or so), it is only common to see this on the very first conv layer that is looking at the input image.

Answer 1

Many recent effective CNN structures use small filters that preserve the spatial resolution, for example the VGG network and the 100-layer residual network.

I think most importantly having the same input and output size allows for simply stacking up more layers without affecting(decreasing) the spatial resolution, so that we can build deeper networks.

Moreover, with such spacial consistency we can, add some operation between the input and output of a set of layers, as in the residual network,

Formally, denoting the desired underlying mapping as $H(x)$, we let the stacked nonlinear layers fit another mapping of $F(x) := H(x)−x$.

or concatenate the output from filters of different sizes, as in the inception network.