Where can I use kernels other than Gaussian (like Cauchy, laplacian) in kernel methods in machine learning? Or maybe in kernel density estimation? In few papers I read that - kernel used doesn't really matter for kernel density estimation but bandwidth of the kernel is the most important factor. But I did not see any mathematical explanation to this and can't completely believe in this. 
 A: I understand your doubts and have also read similar things. The main tenor was to always (or at least unless you know better) use a Gaussian kernel and adjust the bandwidth accordingly. Large bandwidths then will give smooth fits, small bandwidths more wiggly fits which are closer to the data. In the same relation, one sometimes also reads things like "don't use a polynomial kernel because it is degenerate" or "Gaussian kernels map into an infinite space" and are thus considered superior. This is not true in general. If you need to model a cubic function, for instance, take a cubic polynomial kernel. Further, topics like natural language processing rely on polynial kernels (quoting Wikipedia).
The reason why Gaussian kernels are primarily used is probably because they are easy and flexible alike. As you know, in kernelized linear regression you map onto basis functions $\phi_i(\mathbf x) = \kappa(\mathbf x, \mathbf \xi_i)$, where $\kappa$ is the (not necessarily positive-definite) kernel. With Gaussian kernels, you can smoothly vary between very localized and delocalized basis functions. For polynomials, there is no similar parameter, i.e. they live in the whole space.
Summarizing, there is no reason why not to use the Cauchy, Laplacian and other kernels. (Depending on the application, they also have to satisfy properties like the Mercer condition or the reproducing property... I don't know whether they do.) However, and importantly, due to their similarities with a Gaussian, they will likely yield very similar results and I guess you have to search hard for an application where these are clearly superior.
