I'm trying to estimate the probability density function of a real random variable given its iid realizations. What are some of the standard techniques to do this?

One method I have heard of is the kernel density estimation (KDE). When compared to the method of histogram, the KDE is basically assuming that the density can be represented or approximated by the summation of a set of kernels, which seems to me like a smoothness constraint on the density function. Therefore, I'm wondering whether there are some common assumptions on the smoothness of the probability density function when designing estimators for such problems.

I'd greatly appreciate any of your pointers or references. References to recent advances in research in this field are especially welcome.

Original link to this question: https://math.stackexchange.com/questions/2335353/what-are-some-of-the-common-techniques-for-density-estimation/2335378#2335378

  • 2
    $\begingroup$ If the question was already ask and answered, then what exactly do you need from us? $\endgroup$
    – Tim
    Jun 25, 2017 at 18:40
  • $\begingroup$ Thanks for the comment. I have got what I wanted. The most common method is the KDE. The "common assumption" I was looking for was to assume that the pdf belongs to a Holder class, which is then used to prove the vanishing speed of MSE. $\endgroup$ Jun 25, 2017 at 19:47
  • $\begingroup$ So then why do you ask again? $\endgroup$
    – Tim
    Jun 25, 2017 at 20:25
  • $\begingroup$ @Tim : I would surmise that it's because the question is open-ended: There may be many techniques beyond those in the answer already given. $\endgroup$ Jun 25, 2017 at 21:50

1 Answer 1


If you care for techniques with strong theoretical guarantees on sample complexity for arbitrarily low error as well as run-time, you could check out this work by Chan et al. This paper presents an algorithm for learning $p$-piecewise degree-$d$ polynomials, i.e. distributions where the support can be partitioned into $p$ contiguous intervals, and the function is a polynomial of degree at most $d$ on each interval. They give an algorithm to learn such distributions in polynomial time and with near optimal sample complexity. While this might appear to be an esoteric result for practitioners, it turns out that a LOT of distributions can be well approximated by piecewise polynomials. Read the paper for more details.

Histograms has already been mentioned, but I'd like to point out works this (http://www.iliasdiakonikolas.org/papers/histograms-nips.pdf) and this (same url prefix as the previous one upto and including "papers" and then pods-hist.pdf - I'm saying this because apparently I need more reputation to most more than 2 links on an answer) which give strong theoretical guarantees. The former is for estimation in total variation distance and the latter is for estimation in $\ell_2$ distance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.