34
$\begingroup$

I am trying to get a better understanding of kernel density estimation.

Using the definition from Wikipedia: https://en.wikipedia.org/wiki/Kernel_density_estimation#Definition

$ \hat{f_h}(x) = \frac{1}{n}\sum_{i=1}^n K_h (x - x_i) \quad = \frac{1}{nh} \sum_{i=1}^n K\Big(\frac{x-x_i}{h}\Big) $

Let's take $K()$ to be a rectangular function which gives $1$ if $x$ is between $-0.5$ and $0.5$ and $0$ otherwise, and $h$ (window size) to be 1.

I understand that the density is a convolution of two functions, but I am not sure I know how to define these two functions. One of them should (probably) be a function of the data which, for every point in R, tells us how many data points we have in that location (mostly $0$). And the other function should probably be some modification of the kernel function, combined with the window size. But I am not sure how to define it.

Any suggestions?

Bellow is an example R code which (I suspect) replicates the settings I defined above (with a mixture of two Gaussians and $n=100$), on which I hope to see a "proof" that the functions to be convoluted are as we suspect.

# example code:
set.seed(2346639)
x <- c(rnorm(50), rnorm(50,2))
plot(density(x, kernel='rectangular', width=1, n = 10**4))
rug(x)

enter image description here

$\endgroup$
2
  • 4
    $\begingroup$ Your rug at the bottom gives some rough intuition. Imagine each value $x_i$ from $i = 1$ to $n$ is a spike with an associated weight $1/n$. Now smear each spike using the shape and width of your kernel, so that the spike is transformed to take on the same shape and width, with a height such that the area below is $1/n$. Add the results and you have a kernel density estimate. $\endgroup$
    – Nick Cox
    Commented Oct 23, 2013 at 19:57
  • $\begingroup$ Hi Nick, thank you for the comment. This far in the intuition I already got, it is the turning it formally into the form of the convolution which I was curious to see :) (I'm eager to now go through Whuber's answer!) $\endgroup$
    – Tal Galili
    Commented Oct 23, 2013 at 20:07

2 Answers 2

35
+50
$\begingroup$

Corresponding to any batch of data $X = (x_1, x_2, \ldots, x_n)$ is its "empirical density function"

$$f_X(x) = \frac{1}{n}\sum_{i=1}^{n} \delta(x-x_i).$$

Here, $\delta$ is a "generalized function." Despite that name, it isn't a function at all: it's a new mathematical object that can be used only within integrals. Its defining property is that for any function $g$ of compact support that is continuous in a neighborhood of $0$,

$$\int_{\mathbb{R}}\delta(x) g(x) dx = g(0).$$

(Names for $\delta$ include "atomic" or "point" measure and "Dirac delta function." In the following calculation this concept is extended to include functions $g$ which are continuous from one side only.)

Justifying this characterization of $f_X$ is the observation that

$$\eqalign{ \int_{-\infty}^{x} f_X(y) dy &= \int_{-\infty}^{x} \frac{1}{n}\sum_{i=1}^{n} \delta(y-x_i)dy \\ &= \frac{1}{n}\sum_{i=1}^{n} \int_{-\infty}^{x} \delta(y-x_i)dy \\ &= \frac{1}{n}\sum_{i=1}^{n} \int_{\mathbb{R}} I(y\le x) \delta(y-x_i)dy \\ &= \frac{1}{n}\sum_{i=1}^{n} I(x_i \le x) \\ &= F_X(x) }$$

where $F_X$ is the usual empirical CDF and $I$ is the usual characteristic function (equal to $1$ where its argument is true and $0$ otherwise). (I skip an elementary limiting argument needed to move from functions of compact support to functions defined over $\mathbb{R}$; because $I$ only needs to be defined for values within the range of $X$, which is compact, this is no problem.)

The convolution of $f_X(x)$ with any other function $k$ is given, by definition, as

$$\eqalign{ (f_X * k)(x) &= \int_{\mathbb{R}} f_X(x - y) k(y) dy \\ &=\int_{\mathbb{R}} \frac{1}{n}\sum_{i=1}^{n} \delta(x-y-x_i) k(y) dy \\ &= \frac{1}{n}\sum_{i=1}^{n}\int_{\mathbb{R}} \delta(x-y-x_i) k(y) dy \\ &=\frac{1}{n}\sum_{i=1}^{n} k(x_i-x). }$$

Letting $k(x) = K_h(-x)$ (which is the same as $K_h(x)$ for symmetric kernels--and most kernels are symmetric) we obtain the claimed result: the Wikipedia formula is a convolution.

$\endgroup$
12
  • 1
    $\begingroup$ The situation in two dimensions is explained (in more colloquial terms) and illustrated on the GIS site at gis.stackexchange.com/questions/14374/…. $\endgroup$
    – whuber
    Commented Oct 23, 2013 at 20:04
  • 2
    $\begingroup$ Dear Whuber, I just went through and read your answer with delight! Thank you very much for the explanation and details, your answers (this one, and your others in general) are truly inspiring. Yours, Tal $\endgroup$
    – Tal Galili
    Commented Oct 23, 2013 at 20:28
  • 1
    $\begingroup$ I think this is called a Dirac comb. $\endgroup$
    – Elvis
    Commented Oct 24, 2013 at 5:25
  • 1
    $\begingroup$ @Elvis According to Wikipedia, a Dirac comb corresponds to a periodic (and therefore infinite) set of data. Although the machinery is similar, the concept does not appear to apply to this question. $\endgroup$
    – whuber
    Commented Oct 24, 2013 at 5:31
  • 2
    $\begingroup$ @Jan Your understanding is not quite correct. There is no empirical "density" in the sense of a finite continuous measure. The indicator function of the data integrates to zero (whether you use Lebesgue integration or Riemann integration makes no difference). The generalized function $\delta$ is not a function at all: it's a new mathematical object that can be used only within integrals. The empirical distribution is a mathematical object that, when integrated against any integrable function $g,$ returns the sum (over all data $x_i$) of the values $g(x_i).$ $\endgroup$
    – whuber
    Commented May 19, 2019 at 13:41
1
$\begingroup$

Another convenient way of understanding this connection is from the modeling perspective. Recall that if $X$ and $Y$ are independent, then the PDF of $X+Y$ are the convolution of the marginal PDFs. In kernel density estimation (KDE), we may think about the problem via the following model $$ X = X' + \varepsilon,$$ where $X'$ has the uniform discrete distribution over the $n$ observed observations $\{x_1, x_2, \ldots, x_n\}$ with density $$ f(x') = \frac{1}{n} \sum_{i=1}^n \delta(x'- x_i),$$ as provided above by whuber and $\varepsilon$ is a continuous noise variable independent of $X'$. Then the density of $X$ is their convolution as given by the KDE formula. Depending on the PDF of $\varepsilon$, different kernel functions can be used, e.g., Gaussian kernel.

I believe similar explanations can be made for kernel regression and local linear/polynomial regression, where the focus is more on the mean function. Nevertheless, the mean function is manifested by the density, especially if the Gaussian kernel is used.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.