22
$\begingroup$

This has been asked on other SE sites in the context of operating systems and linear algebra, but the same question bugs me regarding kernel methods used in statistics and machine learning.

Often it is said that kernels, e.g. in kernel density estimation or SVMs, represent some kind of similarity, but I don't get where the name 'kernel' comes from and what is the symbolism of it.

So, what's the etymology of kernels, in the context of statistics and machine learning?

To be clear, I'm well aware what a kernel is and the main properties of it, I'm just curious about the name, as I don't see what it has to do with the seed or core as it's mostly defined in dictionaries. Or at least I can't see a deeper meaning than 'the essential part of the method'.

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
16
$\begingroup$

You can find some information on this matter by checking the website for earliest known uses for some of the words of mathematics maintained by mathematician Jeff Miller. You can also find information on the etymology of the word "kernel" in standard dictionary sources.

Writing in French, Fredholm (1903) used the word "noyau" (core) and Hilbert (1904) adopted this term but wrote in German, yielding the German word "kern" (core). These terms were used in the context of writing about integral equations in functional analysis. Shortly after this, Bôcher (1909), writing in English, uses the term "kernel" to refer to the same objects. The term then spreads through the English-language literature on functional analysis, Fourier analysis, and later, probability and statistics.

According to the above-linked dictionary, this word derives from Old-English and Proto-Germanic. It can refer either to a seed, or to the core, center or essence of an object. The linguistic similarity between the German "kern" and "kernel" appears to be due to similar historical derivations. The word "kernel" is alleged to have derived from a hypothesised (reconstructed) Proto-Germanic word "kurną" (corn). So, based on this history, it seems that etymologically, the word "kernel" refers to a seed, core or essence, and is based on the Anglicisation of an old German word for corn.

UPDATE: This answer has been edited heavily to reflect new information that was brought to my attention by users cbeleites and R.M. I initially thought this may have been a recent Anglicisation of the word "kern", but the dictionary sources suggest that the word "kernel" in English is very old. I am not a linguist, so I am merely setting out information from the above sources.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Never heard of Maxime Bôcher. Ar first, I thought maybe it was a typo for Salomon Bochner, who some years later was very big (a giant) in integral kernels, among other things. $\endgroup$
    – Mark L. Stone
    Apr 21, 2018 at 0:05
  • $\begingroup$ That is my fault - I'm not sure how to type the accented o. How did you do it? $\endgroup$
    – Ben
    Apr 21, 2018 at 0:08
  • 1
    $\begingroup$ I copied and pasted from the Wikipedia listing. $\endgroup$
    – Mark L. Stone
    Apr 21, 2018 at 2:03
  • 1
    $\begingroup$ My real point was I knew Bochner was a giant in kernels, but never heard of Bocher.. Bochner was the thesis advisor of Samuel Karlin, from whom I took a course in Total Positivity (totally positive kernels inducing variation diminishing transformations, and all that jazz). $\endgroup$
    – Mark L. Stone
    Apr 21, 2018 at 2:19
  • 1
    $\begingroup$ I'm not sure whether kernel is an Anglicised version of Kern, at least not in the sense of something that happened only a bit more than 100 years ago. I always took it as a translation: merriam-webster.com/dictionary/kernel definition 1 is called "Kern" in German. E.g. for an apple, the seeds = Kerne, whereas core = Kerngehäuse (literally kernel housing). $\endgroup$
    – cbeleites unhappy with SX
    Apr 21, 2018 at 14:49
12
$\begingroup$

A kernel is used a multitude of times in Machine Learning and statistics. A few examples are:

  1. In Support Vector Machines a kernel is a function that maps the data to a higher-dimensional space where the problem becomes linearly solvable (watch this).

  2. Kernel Density Estimation

    Kernel Density Estimation is a non-parametric way to estimate the probability density function of a random variable.

In this context a kernel is simply a weighting function used in kernel density estimation. The last link also has a few more uses of the word kernel in statistics.

  1. In Convolutional Neural Networks, a kernel is a small matrix that is used to perform the convolution between the image and itself. See here.

In all these examples a kernel is a mathematical function that is used for some sort of transformation on your data. The kernel essentially is the constant part of that transformation. Depending on the choice of kernel we use for each transformation we might get a different effect from it.

Improve this answer
$\endgroup$
3
  • 5
    $\begingroup$ Two more examples I've run across: with regards to probability density functions, the "kernel" can refer to a function that is proportional to the density, i.e. for $\beta e^{-\beta x}$, $e^{-\beta x}$ could be considered the kernel. Second, in numerical analysis, the "kernel" can refer to a core function that needs to computed several times during an iterative algorithm. $\endgroup$
    – Cliff AB
    Apr 20, 2018 at 23:01
  • 7
    $\begingroup$ OP is looking for the etymology of kernels. $\endgroup$
    – Franck Dernoncourt
    Apr 20, 2018 at 23:07
  • 2
    $\begingroup$ Does the use of "kernal" to refer to the nullspace in linear algebra have any relation to any of these usages in machine learning? $\endgroup$
    – syntonicC
    Apr 21, 2018 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.