Explain what a kernel is, in plain English

Question

there are several distinct usages:

kernel density estimation
kernel trick
kernel smoothing

Please explain what the "kernel" in them means, in plain English, in your own words

Answer 1

In both statistics (kernel density estimation or kernel smoothing) and machine learning (kernel methods) literature, kernel is used as a measure of similarity. In particular, the kernel function k(x,.) defines the distribution of similarities of points around a given point x. k(x,y) denotes the similarity of point x with another given point y.

Answer 2

There appear to be at least two different meanings of "kernel": one more commonly used in statistics; the other in machine learning.

In statistics "kernel" is most commonly used to refer to kernel density estimation and kernel smoothing. http://en.wikipedia.org/wiki/Kernel_density_estimation

http://en.wikipedia.org/wiki/Kernel_smoothing

A straightforward explanation of kernels in density estimation can be found here: http://school.maths.uwa.edu.au/~duongt/seminars/intro2kde/

In machine learning "kernel" is usually used to refer to the "kernel trick", a method of using a linear classifier to solve a non-linear problem "by mapping the original non-linear observations into a higher-dimensional space".

http://en.wikipedia.org/wiki/Kernel_trick

A simple visualisation might be to imaging that all of class 0 are with radius r of the origin in an x, y plane (class 0: x^2 + y^2 < r^2); and all of class 1 are beyond radius r in that plane (class 1: x^2 + y^2 > r^2). No linear separator is possible, but clearly a circle of radius r will perfectly separate the data. We can transform the data into three dimensional space by calculating three new variables x^2, y^2 and sqrt(2)*x*y. The two classes will now be separable by a plane in this 3 dimensional space. The equation of that optimally separating hyperplane where z1 = x^2, z2 = y^2 and z3 = sqrt(2)*x*y is z1 + z2 = 1, and in this case omits z3. (If the circle is off-set from the origin, the optimal separating hyperplane will vary in z3 as well.) The kernel is the mapping function which calculates the value of the 2-dimensional data in 3-dimensional space.

There are other uses of "kernels" in mathematics, but these seem to be the main ones in statistics: http://en.wikipedia.org/wiki/Kernel_%28mathematics%29

Answer 3

Following up Thylacoleo's example using the circle to explain the kernel trick (i don't have enough reputation to add a comment directly to his answer)

Was there a simple typo in the equation for the separating hyperplane? and it should be z1 + z2 = r^2, instead of z1 + z2 = 1? Or do I misunderstand?

I agree its a nice simple example to illustrate the concept. Thanks. Though the definition of z3 still seems a bit of a mystery, but apparently it doesn't matter for the example centered at the origin.