In Convolutional Neural Networks (CNN), how we can decide number of kernels between input and hidden layer? I have $32\times32$ input image and $5\times5$ convolution. So in the first hidden layer, the feature map size will be $28\times28$. At this link we can see in C1, the number of feature maps is 4 but in C2, the number of feature maps is 12. So how can we calculate number of kernels? Is it arbitrary or is there logic behind this?
 A: It's important not to confuse kernels with feature maps. 
The kernels are the masks used to perform convolution on your input image. The feature maps are the result of the convolution, your new filtered images.
I feel like the answer by @yasin.yazici explains well how to choose the number of feature maps one would like to use at each layer of a CNN, but does not answer the original question : 
"So how can we calculate number of kernels?"
The number of kernels is tipically the same than the number of resulting feature maps. Say we are using same-size convolutions, on images of 32x32 pixels, and 2D kernels of 3x3 elements. We can separate two different cases for more clarity. (also note that in that context, the terms feature map, representation, and channels are equivalent, just as kernel, filter and mask are also synonyms of each other)
First case : 1 to X feature maps :
2D convolution on a single-channel (gray color scale) image from which we would like to build two different representations (2 feature maps) will require two different kernels. We will use two 3x3 kernels to perform convolutions from a 32x32 input image to two different 32x32 feature maps (32x32x1 to 32x32x2).
Second case : Y to X feature maps :
If we aim to build two representations of an RGB image (3 channels) we will still need two kernels, but each kernels will need to have as many channels as the input image. In other words we will need two (3x3x3) kernels, or equivalently we can say that we trained six different (3x3) masks. So we use two 3x3x3 kernels to perform convolutions to go from a 32x32x3 input image to two different 32x32 feature maps (32x32x3 to 32x32x2).
I think this image explains it well (taken from this great gitbook) :

A: Number of kernels are not arbitrary. They can be chosen either intuitively or empirically. Depend on the task, number of kernels in each layer can change significantly. The more complex the dataset you expect networks with more kernels perform better. Intuitively, number of kernel at layer layer expected to bigger in the previous layers, as number of possible combination grow. That is why, in general, first layer kernels are less than mid- high-level ones. 
You can think number of kernels as hyper-parameter and tune them on validation set. What is your dataset CIFAR-10? 
EDIT:
Especially in the first layers this can be seen more clearly. Mid- to High- level features doesn't necessarily follow this trend. For example, in object recognition task, it is expected that number of contours(or object parts) are greater than number of edges, as later layers are combinations of priors. For sure, not all combinations are used because they may not present (or not enough) on input distribution to be caught by network.
A: Well for example if you have a 2x2 convolution, (assuming one black-and-white color channel), then you never need more than 2x2=4 kernels, because with 4 kernels, you have one kernel per pixel, so e.g. it could literally be each pixel getting its own kernel: [[1,0],[0,0]],[[0,1],[0,0]],[[0,0],[1,0]],[[0,0],[0,1]]. Indeed, so long as you don't have any kernels that are simply multiples of other kernels, you can scale and combine them to produce any 2x2 grid of pixels.
So that tells that you want somewhere between 0 and x kernels, where x is the number of parameters in a kernel.  You want your kernels to essentially do lossy compression - to describe the underlying data well but not perfectly. Otherwise, you're not really doing anything.  And if you have 0 kernels, you literally aren't doing anything.
I've seen 16 kernels for a 3x3 convolution with 3 color channels. 3x3x3=27.  16/27 is about 0.6.  That sounds reasonable.  More than 50% coverage, less than 100%.  I'm only getting started myself, with the MINST handwritten digits, but I'm going to try 66% and 75%, and see if that gets me about 90% coverage (meaning 10% residual).
