When people talk about neural networks, what do they mean when they say "kernel size"? Kernels are similarity functions, but what does that say about kernel size?
Deep neural networks, more concretely convolutional neural networks (CNN), are basically a stack of layers which are defined by the action of a number of filters on the input. Those filters are usually called kernels.
For example, the kernels in the convolutional layer, are the convolutional filters. Actually no convolution is performed, but a cross-correlation. The kernel size here refers to the widthxheight of the filter mask.
The max pooling layer, for example, returns the pixel with maximum value from a set of pixels within a mask (kernel). That kernel is swept across the input, subsampling it.
So nothing to do with the concept of kernels in support vector machines or regularization networks. You can think of them as feature extractors.
As you can see above, the kernel, also known as kernel matrix is the function in between and its size, here 3, is the kernel size(sometimes it is called kernel width).
The awesome gif comes from here.