11
$\begingroup$

I am new to kernel density estimation (KDE), but I want to learn about it to help me calculate probabilities of outcomes in sequencing data. I watched this https://www.youtube.com/watch?v=QSNN0no4dSI as my first introduction to the subject.

As the lecturer was going over different kernels, I realized it confused me that KDE is considered non-parametric even when the kernel was being locally parameterized by points within a bandwidth.

Are the standard deviation and arithmetic mean not parameters of KDE when the kernel is the normal distribution?

$\endgroup$
4
  • 1
    $\begingroup$ For the kernel itself the mean is $0$; but we do specify the bandwidth, $h$, which determines the amount of smoothing; it doesn't mean that we have an underlying parametric model for the density $f$ (even though conditionally on the sample, it's modelled as a finite mixture of Gaussians, the number of parameters in the model is not fixed, but changes with sample size) $\endgroup$
    – Glen_b
    Dec 9, 2015 at 21:55
  • $\begingroup$ Splines are also considered non-parametric despite meeting the same issues in modeling. In fact, I wonder if there exists a non-parametric procedure that can't be formulated as a (possibly infinite dimensional) parametric procedure. $\endgroup$
    – AdamO
    Feb 26 at 17:21
  • $\begingroup$ @AdamO Given some of the approximation theory literature I've read, my guess is that if such a non-parametric procedure exists it would have to have an unusual structure (e.g. in terms of geometry or topology). $\endgroup$
    – Galen
    Feb 26 at 17:30
  • $\begingroup$ In a dataset with $n$ observations, there are at least $n+1$ parameters: the $n$ locations and the common bandwidth. Because $n+1$ is unbounded, this is not a parametric problem. $\endgroup$
    – whuber
    Feb 26 at 18:12

4 Answers 4

6
$\begingroup$

The idea of a non-parametric density estimator is that it should be able to approximate any distribution arbitrarily closely for a large enough sample size. The class of distributions defined by the KDE method is parametric; it arguably has $n+1$ parameters for any $n$ data points (the parameters being the data points themselves and the bandwidth parameter), but if one does not accept that the means are parameters then it still has one parameter (the bandwidth). Under an appropriate estimation method for the bandwidth, as $n \rightarrow \infty$ the KDE will converge towards the true distribution of the observed sequence of values for underlying IID data.

$\endgroup$
13
$\begingroup$

Doing a little more reading, I have seen different definitions that change my perspective of non-parametric models. I believed that a non-parametric model/distribution must entirely lack parameters, but some report that non-parametric statistics may or may not have parameters. What distinguishes non-parametric statistics from parametric statistics is that they do not have a fixed or a priori distribution or model structure.

To answer the question, KDE with a normal kernel does not assume that the resulting distribution will have a particular shape, and does not assume a model structure (Ex. the number of parameters). Thus, KDE with a normal kernel is non-parametric.

$\endgroup$
1
  • 1
    $\begingroup$ This is correct - more or less. In KDE you do not assume any parametric (functional) form of the data. You can contrast that with trying to estimate the density if you assume normality - here you directly make a parametric/functional asummption, and the entire shape of the curve depends on it. There will be a lot of difference between one where you assume normality vs. (say) a Chi-square. However, it makes very little difference if you chose a Gaussian kernel or a Epanechnikov kernel for KDE. Both include an important parameter, namely the bandwidth but its still nonparametric fitting. $\endgroup$
    – Repmat
    Dec 9, 2015 at 19:58
7
$\begingroup$

Maybe this picture helps. Actually, the "parametric" kernels only give weights and do not strongly influence (as @conjectures says) the shape of the density estimate, in the sense that different kernels often produce very similar density estimates.

The example shows 20 purple realizations of some mixture of normals (the blue density). The grey bell curves indicate the weights that each observation yields for the points at which we want density estimates ($[-4,4]$ in the picture). A kernel density estimate at some point $x$ then simply consists of stacking the weights at that point on top of each other.

In the picture, we see that at $x=-2.2$ (with the second observation slightly jittered to make both bars fully visible), essentially only two observations contribute weights to the estimate at that point (technically, with a normal kernel, all weights are nonzero, but as the normal tails decay quickly, the weights quickly become negligible). These weights are the magenta and green bars.

That the orange and red curve agree confirms that this handmade approach to constructing a KDE just reproduces what Rs density command does.

enter image description here

Code:

n <- 20

set.seed(2)

# generate data from mixture density
mixture <- rbinom(n,1,.5)
n1 <- sum(mixture)
x <- sort(c(rnorm(n1,-1,.65),rnorm(n-n1,1,.65)))

# plot theoretical density and data
gridpoints = 1024
x_seq <- seq(-4,4,length = gridpoints)
mixture_density <- function(g) .5*dnorm(g,-1,.65)+.5*dnorm(g,1,.65)
plot(x_seq, mixture_density(x_seq), type="l", lwd=2, col="lightblue", ylab="density", xlab="x")

points(x,rep(0,n),col="purple",cex=1.5,lwd=2)
legend('topright', c("true","handmade","canned","weights","observations"), lty=c(rep(1,4),NA), col=c("lightblue","orange","red","grey","purple"),lwd=c(rep(2,3),1,2), pch = c(rep(NA, 4), 1),bg="white")

Bandwidth <- .3 # play around here

# compute weights arising from individual observations
GaussianWeights <- matrix(nrow=n,ncol=length(x_seq))
for (i in 1:n) {
  psi <- (x_seq-x[i])/Bandwidth
  GaussianWeights[i,] <- dnorm(psi)/(n*Bandwidth)
  lines(x_seq,GaussianWeights[i,],col="grey")
  segments(x[i],0,x[i],dnorm(0)/(n*Bandwidth),lty=3,col="grey")
}

segments(-2.2,0,-2.2,dnorm((-2.2-x[2])/Bandwidth)/(n*Bandwidth),col="green",lwd=2)
segments(-2.22,0,-2.22,dnorm((-2.22-x[1])/Bandwidth)/(n*Bandwidth),col="magenta",lwd=2)

HandMadeKernelDensity <- colSums(GaussianWeights)
lines(x_seq,HandMadeKernelDensity,col="orange",lwd=2)

# compare with shipped density
CannedDensity <- density(x,bw=Bandwidth,from=-4,to=4,n=gridpoints)
lines(CannedDensity,col="red",lwd=2,lty=2) # exactly the same wherever it is computed
$\endgroup$
3
$\begingroup$

KDE is about fitting a density function to data. As the result could be an arbitrary curve (from some wide family) it could be considered nonparametric.

I'm not familiar with the video you link but assume the kernel is being used to fit some local part of the overall density function's shape. Therefore the parameters of the kernel function influence but do not dictate the shape of the overall density function which is mainly determined by the data. Contrast this to a regular Normal density function where the parameters absolutely determine the shape of the density.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.