# How to intuitively explain what a kernel is?

Many machine learning classifiers (e.g. support vector machines) allow one to specify a kernel. What would be an intuitive way of explaining what a kernel is?

One aspect I have been thinking of is the distinction between linear and non-linear kernels. In simple terms, I could speak of 'linear decision functions' an 'non-linear decision functions'. However, I am not sure if calling a kernel a 'decision function' is a good idea.

Suggestions?

Kernel is a way of computing the dot product of two vectors $\mathbf x$ and $\mathbf y$ in some (possibly very high dimensional) feature space, which is why kernel functions are sometimes called "generalized dot product".

Suppose we have a mapping $\varphi \, : \, \mathbb R^n \to \mathbb R^m$ that brings our vectors in $\mathbb R^n$ to some feature space $\mathbb R^m$. Then the dot product of $\mathbf x$ and $\mathbf y$ in this space is $\varphi(\mathbf x)^T \varphi(\mathbf y)$. A kernel is a function $k$ that corresponds to this dot product, i.e. $k(\mathbf x, \mathbf y) = \varphi(\mathbf x)^T \varphi(\mathbf y)$.

Why is this useful? Kernels give a way to compute dot products in some feature space without even knowing what this space is and what is $\varphi$.

For example, consider a simple polynomial kernel $k(\mathbf x, \mathbf y) = (1 + \mathbf x^T \mathbf y)^2$ with $\mathbf x, \mathbf y \in \mathbb R^2$. This doesn't seem to correspond to any mapping function $\varphi$, it's just a function that returns a real number. Assuming that $\mathbf x = (x_1, x_2)$ and $\mathbf y = (y_1, y_2)$, let's expand this expression:

\begin{align} k(\mathbf x, \mathbf y) & = (1 + \mathbf x^T \mathbf y)^2 = (1 + x_1 \, y_1 + x_2 \, y_2)^2 = \\ & = 1 + x_1^2 y_1^2 + x_2^2 y_2^2 + 2 x_1 y_1 + 2 x_2 y_2 + 2 x_1 x_2 y_1 y_2 \end{align}

Note that this is nothing else but a dot product between two vectors $(1, x_1^2, x_2^2, \sqrt{2} x_1, \sqrt{2} x_2, \sqrt{2} x_1 x_2)$ and $(1, y_1^2, y_2^2, \sqrt{2} y_1, \sqrt{2} y_2, \sqrt{2} y_1 y_2)$, and $\varphi(\mathbf x) = \varphi(x_1, x_2) = (1, x_1^2, x_2^2, \sqrt{2} x_1, \sqrt{2} x_2, \sqrt{2} x_1 x_2)$. So the kernel $k(\mathbf x, \mathbf y) = (1 + \mathbf x^T \mathbf y)^2 = \varphi(\mathbf x)^T \varphi(\mathbf y)$ computes a dot product in 6-dimensional space without explicitly visiting this space.

Another example is Gaussian kernel $k(\mathbf x, \mathbf y) = \exp\big(- \gamma \, \|\mathbf x - \mathbf y\|^2 \big)$. If we Taylor-expand this function, we'll see that it corresponds to an infinite-dimensional codomain of $\varphi$.

Finally, I'd recommend an online course "Learning from Data" by Professor Yaser Abu-Mostafa as a good introduction to kernel-based methods. Specifically, lectures "Support Vector Machines", "Kernel Methods" and "Radial Basis Functions" are about kernels.

• Current tag definition: "Intuitive: questions that seek a conceptual or non-mathematical understanding of statistics." No clear indication whether conceptual is treated as a synonym of non-mathematical. Jul 8, 2018 at 15:58

# A visual example to help intuition

Consider the following dataset where the yellow and blue points are clearly not linearly separable in two dimensions.

If we could find a higher dimensional space in which these points were linearly separable, then we could do the following:

• Map the original features to the higher, transformer space (feature mapping)
• Perform linear SVM in this higher space
• Obtain a set of weights corresponding to the decision boundary hyperplane
• Map this hyperplane back into the original 2D space to obtain a non linear decision boundary

There are many higher dimensional spaces in which these points are linearly separable. Here is one example

$$x_1, x_2 : \rightarrow z_1, z_2, z_3$$ $$z_1 = \sqrt{2}x_1x_2 \ \ z_2 = x_1^2 \ \ z_3 = x_2^2$$

This is where the Kernel trick comes into play. Quoting the above great answers

Suppose we have a mapping $$\varphi \, : \, \mathbb R^n \to \mathbb R^m$$ that brings our vectors in $$\mathbb R^n$$ to some feature space $$\mathbb R^m$$. Then the dot product of $$\mathbf x$$ and $$\mathbf y$$ in this space is $$\varphi(\mathbf x)^T \varphi(\mathbf y)$$. A kernel is a function $$k$$ that corresponds to this dot product, i.e. $$k(\mathbf x, \mathbf y) = \varphi(\mathbf x)^T \varphi(\mathbf y)$$

If we could find a kernel function that was equivalent to the above feature map, then we could plug the kernel function in the linear SVM and perform the calculations very efficiently.

# Polynomial kernel

It turns out that the above feature map corresponds to the well known polynomial kernel : $$K(\mathbf{x},\mathbf{x'}) = (\mathbf{x}^T\mathbf{x'})^d$$. Let $$d = 2$$ and $$\mathbf{x} = (x_1, x_2)^T$$ we get

\begin{aligned} k(\begin{pmatrix} x_1 \\ x_2 \end{pmatrix}, \begin{pmatrix} x_1' \\ x_2' \end{pmatrix} ) & = (x_1x_1' + x_2x_2')^2 \\ & = 2x_1x_1'x_2x_2' + (x_1x_1')^2 + (x_2x_2')^2 \\ & = (\sqrt{2}x_1x_2, \ x_1^2, \ x_2^2) \ \begin{pmatrix} \sqrt{2}x_1'x_2' \\ x_1'^2 \\ x_2'^2 \end{pmatrix} \end{aligned}

$$k(\begin{pmatrix} x_1 \\ x_2 \end{pmatrix}, \begin{pmatrix} x_1' \\ x_2' \end{pmatrix} ) = \phi(\mathbf{x})^T \phi(\mathbf{x'})$$

$$\phi(\begin{pmatrix} x_1 \\ x_2 \end{pmatrix}) =\begin{pmatrix} \sqrt{2}x_1x_2 \\ x_1^2 \\ x_2^2 \end{pmatrix}$$

# Visualizing the feature map and the resulting boundary line

• Left-hand side plot shows the points plotted in the transformed space together with the SVM linear boundary hyperplane
• Right-hand side plot shows the result in the original 2-D space

### Source

• "f we could find a higher dimensional space in which these points were linearly separable" Why not take a lower-dimensional space e.g. the radius of the points? Apr 4, 2020 at 14:58
• Sure it will work with the mapping z = x^2 +y^2 and since this is even lower-dimensional than the original data, using the kernel view would not make any sense. Therefore your example does not really motivate the use of kernels. But I get that usually, we do not see that easily, which nonlinear mapping to use. Apr 16, 2020 at 12:51
• Are there general rules on how to show that a function is a valid kernel? Jul 15, 2021 at 16:13
• what is $x'$, it's another datapoint in the dataset ? We chose it arbitraly ? Or we compare each point in the dataset with another ? Jul 23, 2021 at 14:40
• @Oliver yes they are called Mercer's theorem, see here for a summary: xavierbourretsicotte.github.io/Kernel_feature_map.html Jul 23, 2021 at 19:54

A very simple and intuitive way of thinking about kernels (at least for SVMs) is a similarity function. Given two objects, the kernel outputs some similarity score. The objects can be anything starting from two integers, two real valued vectors, trees whatever provided that the kernel function knows how to compare them.

The arguably simplest example is the linear kernel, also called dot-product. Given two vectors, the similarity is the length of the projection of one vector on another.

Another interesting kernel examples is Gaussian kernel. Given two vectors, the similarity will diminish with the radius of $\sigma$. The distance between two objects is "reweighted" by this radius parameter.

The success of learning with kernels (again, at least for SVMs), very strongly depends on the choice of kernel. You can see a kernel as a compact representation of the knowledge about your classification problem. It is very often problem specific.

I would not call a kernel a decision function since the kernel is used inside the decision function. Given a data point to classify, the decision function makes use of the kernel by comparing that data point to a number of support vectors weighted by the learned parameters $\alpha$. The support vectors are in the domain of that data point and along the learned parameters $\alpha$ are found by the learning algorithm.

• Dot product and projection are not quite identical. May 18, 2015 at 21:18
• In the case of SVM, I believe that kernels are distance measures in different spaces. This is in keeping with the idea that a SVM generalizes a support vector classifier. In general, kernels can be more complicated.
– meh
May 20, 2015 at 2:56

Very simply (but accurately) a kernel is a weighing factor between two sequences of data. This weighing factor can assign more weight to one "data point" at one "time point" than the other "data point", or assign equal weight or assign more weight to the other "data point" and so on.

This way the correlation (dot product) can assign more "importance" at some points than others and thus cope for non-linearities (e.g non-flat spaces), additional information, data smoothing and so on.

In still another way a kernel is a way to change the relative dimensions (or dimension units) of two data sequences in order to cope with the things mentioned above.

In a third way (related to the previous two), a kernal is a way to map or project one data sequence onto the other in 1-to-1 manner taking into account given information or criteria (e.g curved space, missing data, data re-ordering and so on). So for example a given kernel may stretch or shrink or crop or bend one data sequence in order to fit or map 1-to-1 onto the other.

A kernel can act like a Procrustes in order to "fit best"

• I think you may be talking about kernels in the sense of kernel density estimation, not the positive-semidefinite Mercer kernels used in SVMs and related methods. May 22, 2015 at 5:30
• @Dougal, in the sense of this answer the kernel is a weighing function or measure used to correlate data in a specific manner or to exploit certain data features, so SVM kernel methods are covered as well May 22, 2015 at 7:07

In the top answer to this question, there's a link to the lecture of Pr. Yaser Abu-Mostafa from CalTech and it gives a very nice intuition of it... so I'll try to explain what i understood, without equation:

• a kernel is a function (relatively simple to compute) taking two vectors (living in the X space) and returning a scalar
• that scalar happens in fact to be exactly the dot-product of our two vectors taken to a higher dimension space Z
• so, the kernel tells you how close two vectors are in that Z space, without paying the (possibly enormous) price of computing their coordinates there
• that's all you need to fit an SVM model! in a regular SVM model, you would have used the dot-product in the X space... using the kernel instead is as if you were doing the same thing in the Z space

Here's some words from the lecture:

Think of it this way. I am a guardian of the Z space. I'm closing the door. Nobody has access to the Z space.

You come to me with requests... If you give me an x and ask me, what is the transformation, that's a big demand. I have to hand you a big z. And I may not allow that.

But let's say that all I'm willing to give you are inner products. You give me x and x dash, I close the door, do my thing, and come back with a number, which is the inner product between z and z dash, without actually telling you what z and z dash were. That would be a simple operation.

And if you can get away with it, then that's a pretty good thing.

Selected parts of the lecture:

as was said in comments @ttnphns: "Dot product and projection are not quite identical." – kernel trick is utilized in computations involving the dot products (x, y). kernel itself programmically is just a core of a computer's OS, that controls & distributes "time" & "power" between CPU, RAM, Devices & etc. projection is builded Feature Map, as I understand.

So, defining either kernel (more precise here - certain kernel-trick) to use it in algorithm - you simply define certain type of decomposition to matrices & approximation-type for the learning, But as so as this essentially leads to certain working load for CPU & RAM (as so as the learning process is rather hardware resources consuming) - they called the parameter simply "kernel". (I think, it really is as simple as just the name meaning the direction of pc-resources distribution for certain type of derivation/approximation)