# Explain what a kernel is, in plain English

there are several distinct usages:

kernel density estimation
kernel trick
kernel smoothing

Please explain what the "kernel" in them means, in plain English, in your own words

-
Not to be rude, but isn't this a question that is already answered ad nausea on Wikipedia and the likes? Google gave me the answer within 15 seconds... –  Joris Meys Sep 9 '10 at 8:32
I absolutely hate wikipedia answers for stats. There are rambling, symbolic messes. I am looking for a gem of an answer that can explain the answer in plain English, as I believe that that shows a deeper level of understanding than a math equation. There are many popular "plain English" questions on here, and for good reason. –  Neil McGuigan Sep 9 '10 at 18:04
@NeilMcGuigan - so I am not alone with this observation –  Raffael Nov 18 '13 at 8:37
Excellent question! =) been wondering the same thing! –  jjepsuomi May 7 at 6:52

In both statistics (kernel density estimation or kernel smoothing) and machine learning (kernel methods) literature, kernel is used as a measure of similarity. In particular, the kernel function k(x,.) defines the distribution of similarities of points around a given point x. k(x,y) denotes the similarity of point x with another given point y.

-
This is a nice way of putting it. I am wondering if you can generalize this description to also apply to the kernel of 'kernel density estimation'. –  shabbychef Sep 9 '10 at 16:18
In a way, yes. One way to understand kernel density estimation is that you approximate the density of a point from some distribution as a weighted average of its similarities with a set of points from the distribution. So the notion of similarity does play a role here as well. –  ebony1 Sep 9 '10 at 17:14

There appear to be at least two different meanings of "kernel": one more commonly used in statistics; the other in machine learning.

In statistics "kernel" is most commonly used to refer to kernel density estimation and kernel smoothing.

A straightforward explanation of kernels in density estimation can be found (here).

In machine learning "kernel" is usually used to refer to the kernel trick, a method of using a linear classifier to solve a non-linear problem "by mapping the original non-linear observations into a higher-dimensional space".

A simple visualisation might be to imaging that all of class $0$ are with radius $r$ of the origin in an x, y plane (class $0$: $x^2 + y^2 < r^2$); and all of class $1$ are beyond radius $r$ in that plane (class $1$: $x^2 + y^2 > r^2$). No linear separator is possible, but clearly a circle of radius $r$ will perfectly separate the data. We can transform the data into three dimensional space by calculating three new variables $x^2$, $y^2$ and $\sqrt{2}xy$. The two classes will now be separable by a plane in this 3 dimensional space. The equation of that optimally separating hyperplane where $z_1 = x^2, z_2 = y^2$ and $z_3 = \sqrt{2}xy$ is $z_1 + z_2 = 1$, and in this case omits $z_3$. (If the circle is off-set from the origin, the optimal separating hyperplane will vary in $z_3$ as well.) The kernel is the mapping function which calculates the value of the 2-dimensional data in 3-dimensional space.

In mathematics, there are other uses of "kernels", but these seem to be the main ones in statistics.

-
Very nice! I'm going to use your example with the circle to explain kernel methods, as it is the best visualization I met up til now. Thanks! –  Joris Meys Sep 10 '10 at 13:47
Following up Thylacoleo's example using the circle to explain the kernel trick (i don't have enough reputation to add a comment directly to his answer) Was there a simple typo in the equation for the separating hyperplane? and it should be z1 + z2 = r^2, instead of z1 + z2 = 1? Or do I misunderstand? I agree its a nice simple example to illustrate the concept. Thanks. Though the definition of z3 still seems a bit of a mystery, but apparently it doesn't matter for the example centered at the origin. –  Alex Blakemore Sep 11 '10 at 1:37
Yes there was a typo. Thanks for that Alex. I don't always proofread :-) –  Thylacoleo Sep 13 '10 at 7:26
The following video was proposed by an anonymous potential editor as "a great visualisation of what Thylacoleo explained:" youtube.com/watch?v=3liCbRZPrZA –  gung Jun 6 '13 at 22:56

It's the lumpy bit in the middle (normally) of a distribution

-

This post does not cite any references or sources. Please help improve this post by adding citations to reliable sources. Unsourced material may be challenged and removed.

I have downvoted this--pending some citation--in the belief that this has never been the meaning of "kernel" in the statistical or mathematical literature. I would be very interested and grateful to be proven wrong, though. –  whuber Jul 8 at 18:38