I have iid data-points $x_1, \dots, x_n$, generated by an unknown density $f(x)$. So far I have approximated $f(x)$ with a normal $N(\hat{\mu}, \hat{\sigma}^2 )$, where $\hat{\mu}$ and $\hat{\sigma}^2$ are sample average and variance, but I would like something more flexible.

I have tried with simple Kernel density estimators, but the problem is that in my application I often need to evaluate the estimated density $\hat{f}(x)$ deep in the tails (say I need $\hat{f}(x)$ for $|x| >> 2\sigma$) and it looks like $\hat{f}(x)$ drops to zero very rapidly in the tails.

This image shows what I mean:

Normal vs KDE for normal data Here is the R code:

# Generate normal data
N <- 10000
x <- rnorm(N)

# Calculate Bandwidth
tmp <- density(x, kernel = "gaussian")

# Evaluate the log-density
xSeq <- y <- seq(-5, 5, by = 0.1)

for(ii in 1:length(y))
y[ii] <- log(sum(dnorm(xSeq[ii]-x, 0, sd = tmp$bw))/N)

plot(xSeq, y, type = 'l', main = "Black = KDE, broken = normal (true)",
     ylab = "Log-density", xlab = "x")
lines(xSeq, dnorm(xSeq, log = TRUE), lty = 2)

In this case the true density is really Gaussian, but in my application it is unknown. Obviously estimating tail probabilities is very difficult, but what I would like is a KDE that doesn't drop to zero so fast. In the literature I found a lot about fat-tailed kernels, but I'm not sure whether that's what I need. Thanks!

EDIT: One might ask "why do you need to evaluate the estimated density very far in the tails?". The answer is that I want to use this density to estimate a likelihood $f(x_0|\theta)$, where $x_0$ is an observation and $\theta$ is a unknown parameter. Given a set of parameters $\theta^0$ I simulate $n$ data-points $x_1,...,x_n$, estimate their density and use it to get an estimate of the likelihood $\hat{f}(x_0|\theta^0)$. Give that the real $\theta$ is unknown, when I initialize the algorithm at $\theta^0$ my sample can be very far from $x_0$ and hence I will evaluate the estimated density in the extreme tails.

  • 1
    $\begingroup$ what prior knowledge do you have about the support of the data? It seems like you know what the tails "should" be doing. Tell us more. $\endgroup$ – Memming May 18 '13 at 17:33
  • $\begingroup$ In general the support of the data is the real line, but don't have any particular distribution in mind. I'm not saying that the distribution of the data is heavy tailed, but I would prefer a density estimator that has heavy tails to one that has thin tails like the KDE I plotted. One reason is that I prefer having wider confidence intervals since they are more conservative. In the plot you see that the KDE density collapses suddenly around -4 and +4 and that's not nice because in my application I need to evaluate the density really deep in the tail (like $x = 10$ in this case). $\endgroup$ – Matteo Fasiolo May 18 '13 at 18:04
  • $\begingroup$ You could try mixture of Gaussians to model your density. $\endgroup$ – Memming May 18 '13 at 19:24
  • 3
    $\begingroup$ I'd suggest considering a logspline density estimate $\endgroup$ – Glen_b May 19 '13 at 2:02

I have a number of questions.

Why do you care about the tail of the distribution if you don't know where it is?

You have said that you would need to evaluate the density at x = 10 in your example. How many samples do you have? For KDE I am fairly certain you don't have enough for this. Looking at that part of the tail, we have

$$P(|X| > 10) = erf(\frac{10}{\sqrt{2}})$$

and thus one sample in $\sim 6.5\times10^{22}$ will have $|x| \geq 10$ (wolfram alpha). Which brings me back to the question of why you need to evaluate that far out on the tail.

Your tail density will also be significantly affected by the bandwidth that you select for your KDE. If you make your kernel have bandwidth = 1, I bet your graph will look a lot nicer at the tails, but this is only due to the coincidence of the kernel having the same tail properties as the density of interest.

  • $\begingroup$ Your question is very legitimate, so I have edited my question to add some information about my application. Regarding the number of samples I have: I control the sample size, so I can sample as much as I want, but the computational cost increases. In addition, as you have said, it might take forever before I get a sample in the far tails. $\endgroup$ – Matteo Fasiolo May 22 '13 at 10:50

It may be a consequence of the way you have presented your example, but it looks like your density function has finite support (e.g, a truncated Gaussian)? If this is the case, why not use a spline density estimator with linear tails: http://cran.r-project.org/web/packages/pendensity/vignettes/pendensity.pdf

You could also have a look into wavelet density estimation: http://cran.r-project.org/web/packages/wavethresh/wavethresh.pdf

I'm not sure about the suggestion of using a logspline? Surely this will have by construction exponential (fast decaying) tails?

  • $\begingroup$ Thanks for your answer. Actually in my example the data is normal (in fact x <- rnorm(N)) so the real density has the real line as support. The KDE drops to zero so sharply because there are no sample points outside $x \in [-4,4]$. $\endgroup$ – Matteo Fasiolo May 20 '13 at 22:35
  • $\begingroup$ Perhaps the 'application density' can be assumed to have finite support? You could also just estimate a variance and mean from your data and fit a Gaussian. $\endgroup$ – GeorgeWilson May 20 '13 at 23:58
  • $\begingroup$ No, in general I can't assume that. The normal is what I was using and it is quite robust, but I was looking for something more flexible. $\endgroup$ – Matteo Fasiolo May 22 '13 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.