Kernel Density Estimation - Physical Interpretation?

Question

I just read this article about the motivation for KDE. From what I understand, you are using Gaussian probability density distributions for each datapoint and then, depending on the selected kernel width, you get a KDE curve. I had a few questions:

1. What is the kernel width defined as, for these Gaussian distributions? Is it just the variance?
2. From what I understand, you add the probability densities corresponding to each point to get the full KDE. What does the resulting value curve physically mean? Is it some sort of cumulative probability density?

Answer 1

I think there is no set standard for any of these characteristics of a KDE, For example, the implementation in R gives a choice among seversl shapes of 'kernels' (including 'gaussian' and one of your own). Also, a choice of widths. KDEs in R enclose (almost exactly) unit area.

The R function is density. I have found the default configuration to be about right for many applications. The goal is to estimate the population density from the data. Not surprisingly, it helps to have a large sample size.

If you're estimating the mode of a distribution, the maximum of a KDE is usually better than an attempt to find the mode of a histogram.

References: Perhaps read Wikipedia. Also see Q&A's in the right margin of this page under 'Related/'. Moreover, I have found publications of Bernard Silverman to be especially clear and useful, explaining the theory.

Below are examples using the default KDE in R. In each panel of the figure, the population density of $\mathsf{Gamma}(5, 1/5)$ is plotted as a thin black line, the KDE as a thick red line. (Population mode is 20.)

set.seed(712)
par(mfrow=c(3,1))

x = rgamma(50, 5, 1/5)
  hist(x, prob=T, col="skyblue2", main="n=50")
  rug(x)
  curve(dgamma(x, 5, 1/5), add=T)
  lines(density(x), lwd=2, col="red")

x = rgamma(500, 5, 1/5)
  hist(x, prob=T, col="skyblue2", main="n=500")
  rug(x)
  curve(dgamma(x, 5, 1/5), add=T)
  lines(density(x), lwd=2, col="red")

x = rgamma(5000, 5, 1/5)
  hist(x, prob=T, col="skyblue2", main="n=5000")
  curve(dgamma(x, 5, 1/5), add=T)
  lines(density(x), lwd=2, col="red")
par(mfrow=c(1,1))

Answer 2

1. What is the kernel width defined as, for these Gaussian distributions? Is it just the variance?

In kernel density estimation the width of the kernel densities (called the "bandwidth" in this context) is left as a free parameter and estimated from the data. For the case of a Gaussian KDE the estimated density is:

$$\hat{f}_h(x) = \frac{1}{n h} \sum_{i=1}^n \phi \Big( \frac{x-x_i}{h} \Big),$$

where $\phi$ is the standard normal density and $h>0$ is the "bandwidth". As you can see, this bandwidth parameter is effectively just the standard deviation of the Gaussian kernels used in the density estimator. In practice, this parameter is estimated from the data, which can be done using maximum likelihood estimation, or other methods. Given some estimator $h_*: \mathbb{R}^n \rightarrow \mathbb{R}_+$ you estimate the bandwidth as $h_*(\mathbf{x})$ and so your estimated density is then:

$$\hat{f}(x) \equiv \hat{f}_{h_*(\mathbf{x})}(x) = \frac{1}{n h_*(\mathbf{x})} \sum_{i=1}^n \phi \Big( \frac{x-x_i}{h_*(\mathbf{x})} \Big).$$

2. From what I understand, you add the probability densities corresponding to each point to get the full KDE. What does the resulting value curve physically mean? Is it some sort of cumulative probability density?

You average the kernel density at each point (rather than just summing them) so that you get a density function as your output. The resulting density curve is the estimated density of the distribution that generated the data. The above formula gives a probability density function, not a cumulative distribution, but you can easily get the latter by taking the corresponding average of normal CDFs instead of PDFs.