1
$\begingroup$

Perhaps this question is too broad, but I would like to know - how does one use a kernel density estimate in practice? I know of course that one can use it to draw pretty pictures on top of histograms, but what in particular can a KDE be used for that the empirical distribution itself cannot be used for, especially with the whole notion of needing to properly determine the correct bandwith?

$\endgroup$
3
  • 1
    $\begingroup$ Kernel density estimators are also used in in nonparametric smoothing regressions. See, for example Buja, A., Hastie, T., & Tibshirani, R. (1989). Linear Smoothers and Additive Models. The Annals of Statistics, 17(2), 453–510. $\endgroup$
    – Alexis
    Aug 27, 2021 at 15:44
  • 1
    $\begingroup$ One elementary use is to find the mode of a sample. $\endgroup$
    – BruceET
    Aug 27, 2021 at 16:48
  • $\begingroup$ Rather than only drawing pretty pictures on top of histograms, lose the histogram altogether. That's not to say that histograms should be completely replaced. In other words it seems a bit schizophrenic to show both especially if you think the underlying distribution is relatively smooth and don't have too worry too much about boundaries. $\endgroup$
    – JimB
    Aug 27, 2021 at 16:52

3 Answers 3

2
$\begingroup$

As noticed by others, empirical distribution is not smooth, it's a step function. Say that you observed three datapoints: 2.56, 4.17, and 4.89 and I ask you "what would be the probability density of observing the value 3.14?". Given empirical distribution, there is no nice answer, you need to make some rather arbitrary decision on how to proceed. It's not really that empirical distribution "cannot be used" in such cases, but that it would serve as a very rough approximation of the underlying distribution.

There are many uses of kernel densities, for example:

  • Plotting is an obvious one.
  • Estimating mode of a continuous distribution, as mentioned by @BruceET in the comment.
  • Kernel regression, a non-parametric regression model.
  • Naive Bayes algorithm can use them to approximate the distributions of continuous variables.
  • Kernel discriminant analysis is another classification algorithm using KDE.
  • In cases involving data-based optimization or sampling, you often want not to be blind about the regions of the distribution where the data was not observed. In such cases, kernel density is a nice and simple way to "interpolate" those regions.

etc.

Yes, choosing the bandwidth is somehow arbitrary, but it often does not hurt us that much to pick imperfect bandwidth and there are already many available algorithms for picking it. Notice however that when using other methods (empirical distribution, $k$NN) you run into other problems, like deciding on how to interpolate and extrapolate from the estimated distributions, or picking other kinds of hyperparameters.

$\endgroup$
1
$\begingroup$

Here's one place where KDE is more convenient than ECDF: inverse CDF technique for PRNG. Yes, you can in principle use ECDF too, but since ECDF is step function, its inverse is not continuous either, so you have to do some sort of interpolation between the observations. It's not as elegant as KDE.

$\endgroup$
1
$\begingroup$

Kernel density estimation is a broad topic. I have found publications on this topic by Bernard Silverman to be informative and easy to read.

KDEs are used to approximate probability density functions of continuous distributions from data in one or more dimensions. This Answer is limited to one-dimensional distributions.

One use of KDEs is to make sense of the concept of the mode of a sample. Elementary textbooks have formulas to find the 'mode' of a sample from a histogram. Often, these formulas choose a particular value in the histogram bin with the largest count. One difficulty with this approach is that there can be many different ways in which to choose bins for a histogram and so the 'sample mode' depends on how the histogram is drawn.

Consider the following sample of size $n=500$ from a gamma population distributed $\mathsf{Gamma}(\mathrm{shape}=5, \mathrm{rate}=0.1),$ which has mean $\mu = 5/0.1 = 50$ and mode $\delta = (5-1)/0.1 = 40.$

set.seed(827)
x = rgamma(500, 5, 0.1)
summary(x); length(x);  sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  6.573  35.058  46.678  50.415  62.719 138.549 
[1] 500          # sample size
[1] 21.70053     # sample sd

Two histograms of these data are shown below. The gamma population density curve (solid black) and the default kernel density estimate, KDE, from R (dotted red) are shown. The density and KDE are exactly the same in the two panels. However, respective modal intervals are $(49,60]$ and so $(40,50];$ histogram-based 'sample modes' would differ.

enter image description here

R code for histograms.

par(mfrow=c(1,2))
hist(x, prob=T, br=5, ylim=c(0,0.02),col="skyblue2")
 curve(dgamma(x,5,.1), add=T)
  lines(density(x), lwd=2, lty="dotted", col="red")
hist(x, prob=T, ylim=c(0,0.02),col="skyblue2")
 curve(dgamma(x,5,.1), add=T)
  lines(density(x), lwd=2, lty="dotted", col="red")
par(mfrow=c(1,1))

In R, the KDE connects 512 points, with horizontal and vertical coordinates summarized below.

kde = density(x);  kde

Call:
        density.default(x = x)

Data: x (500 obs.);     Bandwidth 'bw' = 5.361

       x                 y            
 Min.   : -9.508   Min.   :1.785e-06  
 1st Qu.: 31.526   1st Qu.:3.744e-04  
 Median : 72.561   Median :2.738e-03  
 Mean   : 72.561   Mean   :6.086e-03  
 3rd Qu.:113.596   3rd Qu.:1.146e-02  
 Max.   :154.630   Max.   :2.042e-02

The x-value with the largest y-value can be found as follows:

dx = density(x)$x; dy = density(x)$y
mean(dx[dy==max(dy)])
[1] 43.17016

It seems reasonable to say that the sample mode is $43.17.$

Because gamma distributions are continuous, samples will not have tied observations, except those due to rounding. So it may be reasonable to view the mode of sample from a continuous distribution as the value that best approximates the mode of the population distribution.

Of course, you might get slightly different values for the sample mode by varying the bandwidth of the KDE or the types of curves chosen for the kernels. However, the default choices in R seem useful for finding a reasonable value for the sample mode.

Note: The KDE method of finding sample modes does not work well when the mode of the population distribution is at a boundary of the support of the distribution. (For example, an exponential distribution.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.