I came across the following passage in http://cs231n.github.io/convolutional-networks/ with regards to filter sizes in CNNs. Purely because i have seen a number of networks with 5*5 conv filters without 2 padding - i wanted to check if this indeed is best practice. Any thoughts much appreciated.
The conv layers should be using small filters (e.g. $3\times 3$ or at most $5\times 5$), using a stride of $S=1$ , and crucially, padding the input volume with zeros in such way that the conv layer does not alter the spatial dimensions of the input. That is, when $F=3$ , then using $P=1$ will retain the original size of the input. When $F=5$ , $P=2$. For a general $F$, it can be seen that $P=(F−1)/2$ preserves the input size. If you must use bigger filter sizes (such as $7\times7$ or so), it is only common to see this on the very first conv layer that is looking at the input image.