What are some of the common techniques for density estimation? I'm trying to estimate the probability density function of a real random variable given its iid realizations. What are some of the standard techniques to do this?
One method I have heard of is the kernel density estimation (KDE). When compared to the method of histogram, the KDE is basically assuming that the density can be represented or approximated by the summation of a set of kernels, which seems to me like a smoothness constraint on the density function. Therefore, I'm wondering whether there are some common assumptions on the smoothness of the probability density function when designing estimators for such problems.
I'd greatly appreciate any of your pointers or references. References to recent advances in research in this field are especially welcome.
Original link to this question: https://math.stackexchange.com/questions/2335353/what-are-some-of-the-common-techniques-for-density-estimation/2335378#2335378
 A: If you care for techniques with strong theoretical guarantees on sample complexity for arbitrarily low error as well as run-time, you could check out this work by Chan et al. This paper presents an algorithm for learning $p$-piecewise degree-$d$ polynomials, i.e. distributions where the support can be partitioned into $p$ contiguous intervals, and the function is a polynomial of degree at most $d$ on each interval. They give an algorithm to learn such distributions in polynomial time and with near optimal sample complexity. While this might appear to be an esoteric result for practitioners, it turns out that a LOT of distributions can be well approximated by piecewise polynomials. Read the paper for more details.
Histograms has already been mentioned, but I'd like to point out works this (http://www.iliasdiakonikolas.org/papers/histograms-nips.pdf) and this (same url prefix as the previous one upto and including "papers" and then pods-hist.pdf - I'm saying this because apparently I need more reputation to most more than 2 links on an answer) which give strong theoretical guarantees. The former is for estimation in total variation distance and the latter is for estimation in $\ell_2$ distance.
