I am trying to get a better understanding of kernel density estimation.

Using the definition from Wikipedia: https://en.wikipedia.org/wiki/Kernel_density_estimation#Definition

$ \hat{f_h}(x) = \frac{1}{n}\sum_{i=1}^n K_h (x - x_i) \quad = \frac{1}{nh} \sum_{i=1}^n K\Big(\frac{x-x_i}{h}\Big) $

Let's take $K()$ to be a rectangular function which gives $1$ if $x$ is between $-0.5$ and $0.5$ and $0$ otherwise, and $h$ (window size) to be 1.

I understand that the density is a convolution of two functions, but I am not sure I know how to define these two functions. One of them should (probably) be a function of the data which, for every point in R, tells us how many data points we have in that location (mostly $0$). And the other function should probably be some modification of the kernel function, combined with the window size. But I am not sure how to define it.

Any suggestions?

Bellow is an example R code which (I suspect) replicates the settings I defined above (with a mixture of two Gaussians and $n=100$), on which I hope to see a "proof" that the functions to be convoluted are as we suspect.

# example code:
x <- c(rnorm(50), rnorm(50,2))
plot(density(x, kernel='rectangular', width=1, n = 10**4))

enter image description here

  • 3
    $\begingroup$ Your rug at the bottom gives some rough intuition. Imagine each value $x_i$ from $i = 1$ to $n$ is a spike with an associated weight $1/n$. Now smear each spike using the shape and width of your kernel, so that the spike is transformed to take on the same shape and width, with a height such that the area below is $1/n$. Add the results and you have a kernel density estimate. $\endgroup$ – Nick Cox Oct 23 '13 at 19:57
  • $\begingroup$ Hi Nick, thank you for the comment. This far in the intuition I already got, it is the turning it formally into the form of the convolution which I was curious to see :) (I'm eager to now go through Whuber's answer!) $\endgroup$ – Tal Galili Oct 23 '13 at 20:07

Corresponding to any batch of data $X = (x_1, x_2, \ldots, x_n)$ is its "empirical density function"

$$f_X(x) = \frac{1}{n}\sum_{i=1}^{n} \delta(x-x_i).$$

Here, $\delta$ is a "generalized function." Despite that name, it isn't a function at all: it's a new mathematical object that can be used only within integrals. Its defining property is that for any function $g$ of compact support that is continuous in a neighborhood of $0$,

$$\int_{\mathbb{R}}\delta(x) g(x) dx = g(0).$$

(Names for $\delta$ include "atomic" or "point" measure and "Dirac delta function." In the following calculation this concept is extended to include functions $g$ which are continuous from one side only.)

Justifying this characterization of $f_X$ is the observation that

$$\eqalign{ \int_{-\infty}^{x} f_X(y) dy &= \int_{-\infty}^{x} \frac{1}{n}\sum_{i=1}^{n} \delta(y-x_i)dy \\ &= \frac{1}{n}\sum_{i=1}^{n} \int_{-\infty}^{x} \delta(y-x_i)dy \\ &= \frac{1}{n}\sum_{i=1}^{n} \int_{\mathbb{R}} I(y\le x) \delta(y-x_i)dy \\ &= \frac{1}{n}\sum_{i=1}^{n} I(x_i \le x) \\ &= F_X(x) }$$

where $F_X$ is the usual empirical CDF and $I$ is the usual characteristic function (equal to $1$ where its argument is true and $0$ otherwise). (I skip an elementary limiting argument needed to move from functions of compact support to functions defined over $\mathbb{R}$; because $I$ only needs to be defined for values within the range of $X$, which is compact, this is no problem.)

The convolution of $f_X(x)$ with any other function $k$ is given, by definition, as

$$\eqalign{ (f_X * k)(x) &= \int_{\mathbb{R}} f_X(x - y) k(y) dy \\ &=\int_{\mathbb{R}} \frac{1}{n}\sum_{i=1}^{n} \delta(x-y-x_i) k(y) dy \\ &= \frac{1}{n}\sum_{i=1}^{n}\int_{\mathbb{R}} \delta(x-y-x_i) k(y) dy \\ &=\frac{1}{n}\sum_{i=1}^{n} k(x_i-x). }$$

Letting $k(x) = K_h(-x)$ (which is the same as $K_h(x)$ for symmetric kernels--and most kernels are symmetric) we obtain the claimed result: the Wikipedia formula is a convolution.

| cite | improve this answer | |
  • 1
    $\begingroup$ The situation in two dimensions is explained (in more colloquial terms) and illustrated on the GIS site at gis.stackexchange.com/questions/14374/…. $\endgroup$ – whuber Oct 23 '13 at 20:04
  • 2
    $\begingroup$ Dear Whuber, I just went through and read your answer with delight! Thank you very much for the explanation and details, your answers (this one, and your others in general) are truly inspiring. Yours, Tal $\endgroup$ – Tal Galili Oct 23 '13 at 20:28
  • 1
    $\begingroup$ I think this is called a Dirac comb. $\endgroup$ – Elvis Oct 24 '13 at 5:25
  • 1
    $\begingroup$ @Elvis According to Wikipedia, a Dirac comb corresponds to a periodic (and therefore infinite) set of data. Although the machinery is similar, the concept does not appear to apply to this question. $\endgroup$ – whuber Oct 24 '13 at 5:31
  • 1
    $\begingroup$ @Jan Your understanding is not quite correct. There is no empirical "density" in the sense of a finite continuous measure. The indicator function of the data integrates to zero (whether you use Lebesgue integration or Riemann integration makes no difference). The generalized function $\delta$ is not a function at all: it's a new mathematical object that can be used only within integrals. The empirical distribution is a mathematical object that, when integrated against any integrable function $g,$ returns the sum (over all data $x_i$) of the values $g(x_i).$ $\endgroup$ – whuber May 19 '19 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.