Tag Info

How to intuitively explain what a kernel is?

A visual example to help intuition Consider the following dataset where the yellow and blue points are clearly not linearly separable in two dimensions. If we could find a higher dimensional space ...
Accepted

What is the rationale of the Matérn covariance function?

In addition to @Dahn's nice answer, I thought I would try to say a little bit more about where the Bessel and Gamma functions come from. One starting point for arriving at the covariance function is ...
• 3,016
Accepted

Is there any supervised-learning problem that (deep) neural networks obviously couldn't outperform any other methods?

Here is one theoretical and two practical reasons why someone might rationally prefer a non-DNN approach. The No Free Lunch Theorem from Wolpert and Macready says We have dubbed the associated ...
• 21.1k
Accepted

Nystroem Method for Kernel Approximation

Let's derive the Nyström approximation in a way that should make the answers to your questions clearer. The key assumption in Nyström is that the kernel function is of rank $m$. (Really we assume that ...
• 24.8k

Is there any supervised-learning problem that (deep) neural networks obviously couldn't outperform any other methods?

Somewhere on this playlist of lectures by Geoff Hinton (from his Coursera course on neural networks), there's a segment where he talks about two classes of problems: Problems where noise is the key ...
• 5,387

What is the rationale of the Matérn covariance function?

I do not know, but I found this question very interesting and here's what I got after a bit of reading on it. For certain values of $\nu$, the Matérn covariance function can be expressed as a product ...
• 618

• 19.1k
Accepted

• 24.8k
Accepted

Kernels in Gaussian Processes

Notation / setting We are considering a GP regression model: $$y_i = f(x_i) + \epsilon_i$$ where $y_i\in \mathbb{R}$,$x_i \in \mathbb{R}^d$, $f$ a Gaussian process (whose ...
• 7,923
Accepted

Why does a Gaussian Process need to have a PSD kernel? Can I use a non-PSD kernel?

Say that $X \sim \mathcal{GP}(m(\cdot), k(\cdot, \cdot))$. If $k$ is not a PSD kernel, then there is some set of $n$ points $\{ t_i \}_{i=1}^n$ and corresponding weights $\alpha_i \in \mathbb R$ such ...
• 24.8k
Accepted

• 78.8k
Accepted

• 13.4k

Does Mercer's theorem work in reverse?

Does Mercer's theorem work in reverse? Not in all cases. Wikipedia: "In mathematics, specifically functional analysis, Mercer's theorem is a representation of a symmetric positive-definite function ...
• 2,100
Accepted

Is a kernel function basically just a mapping?

My initial understanding is that a kernel is essentially just a mapping into a higher dimension. No. Kernel is a function that calculates dot product in the image of this mapping. It can be ...
• 5,786