5
$\begingroup$

I am currently following the book Gaussian Processes for Machine Learning by C.E. Rasmussen and C.K.I. Williams and I have come across various kernels in their Chapter 4

I have also gone through the kernel cookbook which is a nice description of various basic kernels available and how we can create new kernels from adding, multiplying, and convoluting these basic kernels.

Since we know that regression in a Gaussian process hinges on the choice of kernels, I want to ask if there are basic kernels other than the following and what kind of functions do they draw?

  1. Constant : $\sigma _{o}^{2}$

  2. Linear kernel : $k_{\textrm{Lin}}(x, x') = (x - c)(x' - c)$

  3. Squared exponential kernel : $k_{\textrm{SE}}(x, x') = \sigma^2\exp\left(-\frac{(x - x')^2}{2\ell^2}\right)$ (subset of 7)

  4. Matern : $\frac{1}{{2}^{\nu -1}\Gamma (\nu )}{\left( \frac{\sqrt{2\nu }}{l}r\right)}^{\nu }{K}_{\nu }\left( \frac{\sqrt{2\nu }}{l}r\right)$

  5. Rational quadratic kernel : $k_{\textrm{RQ}}(x, x') = \sigma^2 \left( 1 + \frac{(x - x')^2}{2 \alpha \ell^2} \right)^{-\alpha}$ (subset of 4)

  6. Exponential : $\exp\left( - \frac{r}{l}\right)$ (subset of 7)

  7. $\gamma $ - Exponential : $\exp\left( -{\left(\frac{r}{l}\right)}^{\gamma }\right) $

  8. Periodic kernel : $k_{\textrm{Per}}(x, x') = \sigma^2\exp\left(-\frac{2\sin^2(\pi|x - x'|/p)}{\ell^2}\right)$

  9. Neural networks

$\endgroup$
  • 1
    $\begingroup$ Neural networks are not a kernel. $\endgroup$ – Marc Claesen Dec 18 '14 at 17:20
  • $\begingroup$ @MarcClaesen I suppose the question refers to the so-called neural network covariance function (see, e.g., the linked chapter 4 of the Rasmussen & Williams book. $\endgroup$ – Juho Kokkala Dec 18 '14 at 17:56
  • $\begingroup$ @JuhoKokkala Yes I was refering to the neural network covariance function. $\endgroup$ – Ankit Chiplunkar Dec 18 '14 at 20:17
3
$\begingroup$

Neural networks are not a kernel, they are a learning algorithm.

Plenty of kernel functions exist, such as:

  • sigmoid, popular in the early days of kernel methods due to their influence in neural networks; not really used heavily now
  • Tanimoto/Jaccard/diffusion, popular for binary features
  • tree/graph kernels, popular in natural language processing
  • histogram kernel, popular in image processing -- essentially it's a very fast approximation to the RBF kernel

The right kernel depends very much on the nature of the data. Often the best kernel is a custom-made one, particularly in bioinformatics. The Gaussian/RBF and linear kernels are by far the most popular ones, followed by the polynomial one.

$\endgroup$
  • $\begingroup$ Thanks, as mentioned above I was refering to the neural network covariance/kernel as mentioned in the above book. Also I wanted this question to become some sort of a repository of all the different basic kernels we have/know so far. $\endgroup$ – Ankit Chiplunkar Dec 18 '14 at 20:19
  • $\begingroup$ @AnkitChiplunkar unfortunately, questions regarding enumerations like this one are regularly being closed on cross-validated. $\endgroup$ – Marc Claesen Dec 18 '14 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.