I'm calculating a KDE of one parameter (y, particle density) in bins of another parameter (x, distance from the origin). At small x I have poorly sampled data (10s to 1000s of points per x-bin) while at large x values it is very well sampled (millions of points per x-bin). At small radii, using a KDE seems very important/effective, while at large radii the result is effectively identical to a histogram, but it extremely slow to compute (at least using scipy in python) *[1]. Ultimately I don't need the KDE per se, I just need the smoothed/sampled PDF it produces (i.e. on a regular grid).

It seems like a hybrid approach would be possible in which the KDE is used when the sampling is sparse, but I revert to simple binning when it is very well sampled. Is there a standard procedure for hybridizing these approaches? Or are there techniques for adaptively choosing the bandwidth such that I can use a kernel with finite support that shrinks as the sampling becomes more dense?

*[1] Ultimately, I assume this is because I'm using Gaussian kernels with infinite support, and thus N*M evaluations (for N particles in the given KDE, which is resampled onto a grid of M points).

  • $\begingroup$ A simple procedure would be to compute the KDE of a nonlinear transform of the parameter and then convert it back. Assuming anything close to a slowly-varying density near the origin, a good choice would be the $n^\text{th}$ root of the distance in $n$ dimensions. $\endgroup$ – whuber Feb 1 at 22:41
  • $\begingroup$ @whuber can you expand a bit? I don't see how a transform helps the computational costs. Or is the idea that you reshape the parameter space, to mimic the effect of changing the kernel (/ bandwidth)? Do you know any references/tutorials for this, offhand? $\endgroup$ – DilithiumMatrix Feb 2 at 15:09
  • $\begingroup$ An appropriate transformation will place approximately the same numbers of data in each one of a regularly-spaced set of bins. Some examples appear at stats.stackexchange.com/questions/65866 $\endgroup$ – whuber Feb 2 at 16:21

In case you have coarse bins I would consider to use the method described in the following paper: Efficient Estimation of Smooth Distributions From Coarsely Grouped Data. In appendix 2 you will find a R code illustrating the method. It should be easy to translate in in Python. I hope it helps.

  • $\begingroup$ Thanks for the recommendation, but I don't quite follow how my problem is analogous to "ungrouping", can you elaborate a little on the connection? $\endgroup$ – DilithiumMatrix Feb 4 at 15:06
  • $\begingroup$ Hi I think I misunderstood your question then, sorry. I thought that the under-sampled set of your data resulted in coarser bins. Anyway, the approach in the paper I suggested, is more general and can be used also in the case you do not have coarse bins as in Ill-posed problems with counts, the composite link model and penalized likelihood $\endgroup$ – Gi_F. Feb 4 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.