1
$\begingroup$

Density plots are useful in confirming the fit of a distribution or assessing which distribution to try in order to give the best fit. However, when looking at the tails on log scales, especially for long-tailed datasets, the fit becomes increasingly hard to confirm visually, due to gaps resulting in large downward spikes toward negative infinity. distribution fit on log scale

Are there any techniques in R (or in general) that can accommodate the sparse data points at the tails and allow for smoother density plots in these tail regions? Ideally something where the sd of the kernel increases as the distance from the mean increase.

$\endgroup$
1
  • 2
    $\begingroup$ Tails are tricky; the more outside knowledge / information you can bring to bear, the easier you can make your task. You might perhaps consider log-spline density estimates in place of kernels. Something that can sometimes work well (but in some other circumstances will not be suitable) is applying a transformation to a considerably less heavy-tailed distribution, using a kernel and transforming the resulting density estimate back (don't forget the Jacobian of the transformation); which transformations you might consider depends on the circumstances. That will increase bandwidth in the tail $\endgroup$
    – Glen_b
    Sep 20, 2017 at 2:16

1 Answer 1

0
$\begingroup$

So going off Glen_b's suggestion, I found a logspline package for r. I can't pretend to understand how the splines are calculated (beyond the wikipedia entry), but they do seem to provide a better fit in the tail regions of a distribution.

Here's an example of a fit to 100,000 t distributed points with 3 degrees of freedom: logspline t fit The red line is the generating function, the black line the density plot, and the blue line is the logspline fit.

  library(logspline)
r<-rt(100000,3)
L<-logspline(r,1.3*min(r),max(r)*1.3)
plot.logspline(L,xlim=c(1.4*min(r),max(r)*1.3),log="y",col="4")
lines(density(r),col="1")
x<-seq(1.3*min(r),max(r)*1.3,by=0.01)
y<-dt(x,3)
lines(x,y,col="2") 

A couple of observations:

-The range of the logspline fit needs to be slightly larger than the range of the data, otherwise you'll get a weird inflection at the endpoints. I use a factor of 1.2 to 1.4.

-At the extremes, the lines will always become straight. This is a feature of the splines, not of the data. It's important to keep this in mind because otherwise it may fool you into thinking the tails are decreasing exponentially when they aren't (see right side of picture above).

-As such, while logsplines do increase the range at which you can visually fit the data, they also break down at extreme ranges and cannot be relied on.

-It seems to have trouble with highly concentrated data sets. I have one with ~180000 data points between 0 and 0.5 (most of which are at single particular values) and another ~500 between -6 and 6, and the package just can't deal with the data adequately. I can get the logspline function to run when a little bit of gaussian jitter is added, but the fit is weak. Otherwise the package errors out.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.