# Estimating PDF of continuous distribution from (few) data points

What are some good, established methods for estimating the probability density function (denoted $f(x)$ from here on) of a continuous distribution, given a sample of points $x_1, \ldots, x_n$ drawn from it? I primarily need the PDF for plotting purposes.

The naive approach would be using a histogram, i.e. counting how many points fall into different $[a,b)$ intervals. But this has several problems:

• It doesn't give us $f(\frac{a+b}{2})$, but $\int_a^b f(x) \, dx$, which is not the same, and might look qualitatively different on a plot (e.g. for a Pareto distribution it gives an estimate of the PDF that is not a straight line on log-log scale, this is what I mean by looking qualitatively different).
• It heavily depends on binning, requiring a careful choice of bin size.
• Depending on the distribution, it may require a manual choice of a non-uniform bin size to get something reasonably-looking (e.g. a Pareto distribution requires increasing bins).

I am mainly interested in established methods (please note that I'm not a statistician, I don't have formal training in this, so I may not know about the obvious!), but any ideas are welcome too. E.g. would estimating the CDF by sorting the points, then somehow taking the derivative work? But then the problem is transformed to estimating the derivative of noisy data which is again a difficult problem.

I need this mainly not for fitting the PDF to some function, but for visualizing it.

EDIT: I am in particular interested in techniques that work well for long-tail distributions.

• For visualizing long tails, plotting the cumulative density is often recommended over the PDF. See Power law distributions in empirical data (Clauset et al. 2009) for some examples. It is straight forward to plot the ECDF (e.g. as unconnected points) against the fitted CDF to visualize the goodness of fit. – Andy W Aug 27 '13 at 14:15
• @AndyW At the time I posted this question I was already familiar with the paper you cite, which BTW advocates using maximum likelihood methods, and has less discussion of the CDF. But the question is specifically about estimating the PDF, not the CDF. – Szabolcs Aug 27 '13 at 16:21
• Sorry, my comment was in regards to I need this mainly not for fitting the PDF to some function, but for visualizing it. not to the main title in the question. I point to the power-law paper simply as it has a bunch of illustrations of the types of plots I talk about (and of course related discussion about estimating the distributions themselves). I'm not sure it is useful to say I want to estimate the PDF not the CDF. You estimate parameters of the distribution (through maximum likelihood or whatever). You can then plot the PDF or the CDF of that estimated distribution. – Andy W Aug 27 '13 at 16:52

What you are looking for is kernel density estimation. You should find numerous hits on an internet search for these terms, and it is even on Wikipedia so that should get you started. If you have R at your disposition, the function density provides what you need:

histAndDensity<-function(x, ...)
{
retval<-hist(x, freq=FALSE, ...)
lines(density(x, na.rm=TRUE), col="red")
invisible(retval)
}

• Thanks for the useful pointer! I'll need to read through this properly, but after a superficial look, I have the impression that it also works with a characteristic scale ("smoothing bandwidth"), perhaps similar to the bin-width in the case of histograms. Isn't it going to be plagued by the same problem as fixed-width binning when applying it to a long-tail distribution (i.e. the no points fall into the majority of large-value bins)? – Szabolcs Jul 12 '11 at 8:33
• Here's an example of trying to apply it to a long-tail distribution: goo.gl/o9Muo I just used Mathematica's built-in support for it. In this particular example it doesn't seem to work well for values between 10-20 and at all for anything above 20. Perhaps this is only because of using the default parameters. Notebook attached to PDF. – Szabolcs Jul 12 '11 at 9:00
• @Szabolcs, your question is a general one, so the answer was general also. For specific class of distributions, different methods are usually applied. Long-tailed distributions are for example notorious for their complicated estimation. If you know the functional form of pdf, then the most widely used method for estimating the parameters of the pdf is maximum likelihood. – mpiktas Jul 12 '11 at 9:07
• @mpiktas And of course it was a very useful general answer. Just trying to go further and find more specific ones that are better for my problem. I already found that maximum likelihood is considered a good way to fit these distributions, but right now I'd just like to visualize them somehow (not fit). Perhaps I should update the question mentioning that I'm interested in long-tail ones. – Szabolcs Jul 12 '11 at 9:10
• Long tails and few data points simply do not mix well. As the data points are going to be sparse wrt the functional form, there is no way to distinguish between different functional forms... – Nick Sabbe Jul 12 '11 at 9:18