# What is a “kernel” in plain English?

There are several distinct usages:

• kernel density estimation
• kernel trick
• kernel smoothing

Please explain what the "kernel" in them means, in plain English, in your own words.

-
Not to be rude, but isn't this a question that is already answered ad nausea on Wikipedia and the likes? Google gave me the answer within 15 seconds... –  Joris Meys Sep 9 '10 at 8:32
I absolutely hate wikipedia answers for stats. There are rambling, symbolic messes. I am looking for a gem of an answer that can explain the answer in plain English, as I believe that that shows a deeper level of understanding than a math equation. There are many popular "plain English" questions on here, and for good reason. –  Neil McGuigan Sep 9 '10 at 18:04

In both statistics (kernel density estimation or kernel smoothing) and machine learning (kernel methods) literature, kernel is used as a measure of similarity. In particular, the kernel function k(x,.) defines the distribution of similarities of points around a given point x. k(x,y) denotes the similarity of point x with another given point y.

-
This is a nice way of putting it. I am wondering if you can generalize this description to also apply to the kernel of 'kernel density estimation'. –  shabbychef Sep 9 '10 at 16:18
In a way, yes. One way to understand kernel density estimation is that you approximate the density of a point from some distribution as a weighted average of its similarities with a set of points from the distribution. So the notion of similarity does play a role here as well. –  ebony1 Sep 9 '10 at 17:14
I understand "kernel" in statistics to be borrowed originally from jargon used in discussion of integral equations. –  Nick Cox Sep 16 '14 at 14:47

There appear to be at least two different meanings of "kernel": one more commonly used in statistics; the other in machine learning.

In statistics "kernel" is most commonly used to refer to kernel density estimation and kernel smoothing.

A straightforward explanation of kernels in density estimation can be found (here).

In machine learning "kernel" is usually used to refer to the kernel trick, a method of using a linear classifier to solve a non-linear problem "by mapping the original non-linear observations into a higher-dimensional space".

A simple visualisation might be to imaging that all of class $0$ are with radius $r$ of the origin in an x, y plane (class $0$: $x^2 + y^2 < r^2$); and all of class $1$ are beyond radius $r$ in that plane (class $1$: $x^2 + y^2 > r^2$). No linear separator is possible, but clearly a circle of radius $r$ will perfectly separate the data. We can transform the data into three dimensional space by calculating three new variables $x^2$, $y^2$ and $\sqrt{2}xy$. The two classes will now be separable by a plane in this 3 dimensional space. The equation of that optimally separating hyperplane where $z_1 = x^2, z_2 = y^2$ and $z_3 = \sqrt{2}xy$ is $z_1 + z_2 = 1$, and in this case omits $z_3$. (If the circle is off-set from the origin, the optimal separating hyperplane will vary in $z_3$ as well.) The kernel is the mapping function which calculates the value of the 2-dimensional data in 3-dimensional space.

In mathematics, there are other uses of "kernels", but these seem to be the main ones in statistics.

-
Very nice! I'm going to use your example with the circle to explain kernel methods, as it is the best visualization I met up til now. Thanks! –  Joris Meys Sep 10 '10 at 13:47
Following up Thylacoleo's example using the circle to explain the kernel trick (i don't have enough reputation to add a comment directly to his answer) Was there a simple typo in the equation for the separating hyperplane? and it should be z1 + z2 = r^2, instead of z1 + z2 = 1? Or do I misunderstand? I agree its a nice simple example to illustrate the concept. Thanks. Though the definition of z3 still seems a bit of a mystery, but apparently it doesn't matter for the example centered at the origin. –  Alex Blakemore Sep 11 '10 at 1:37
Yes there was a typo. Thanks for that Alex. I don't always proofread :-) –  Thylacoleo Sep 13 '10 at 7:26
The following video was proposed by an anonymous potential editor as "a great visualisation of what Thylacoleo explained:" youtube.com/watch?v=3liCbRZPrZA –  gung Jun 6 '13 at 22:56

It's the lumpy bit in the middle (normally) of a distribution

-

Some of the information contained in this post requires additional references. Please edit to add citations to reliable sources that support the assertions made here. Unsourced material may be disputed or deleted.

I have downvoted this--pending some citation--in the belief that this has never been the meaning of "kernel" in the statistical or mathematical literature. I would be very interested and grateful to be proven wrong, though. –  whuber Jul 8 '14 at 18:38