9
$\begingroup$

I have some data and was trying to fit a smooth curve to it. However, I do not want to enforce too many prior beliefs or too strong pre-conceptions (except the ones implied by the rest of my question) on it, or any specific distributions.

I just wanted to fit it with some smooth curve (or have a good estimate of the probability distribution that it might have come from). The only method that I know for doing this is kernel density estimation (KDE). I was wondering, if people knew of other methods for estimating such a thing. I just wanted a list of them and from that I can do my own research to find out which ones I want to use.

Giving any links or good references (or intuitions on which ones are good) are always welcome (and are encouraged)!

Improve this question
$\endgroup$
6
  • 4
    $\begingroup$ "I did not want to enforce any prior believes on it" -- then you can't assume it's smooth, or even continuous (those would be prior beliefs). In which case the ecdf is about your only recourse. $\endgroup$
    – Glen_b
    Apr 8, 2014 at 23:28
  • 1
    $\begingroup$ To strong of a believe my be a better way of phrasing my question. I meant I don't want to assume its say, Bernoulli or something that might be to restrictive. I don't know what ecdf is btw. If you have a good suggestion or list of suggestions, feel free to post it. $\endgroup$
    – Charlie Parker
    Apr 8, 2014 at 23:30
  • $\begingroup$ I have updated my question. Is that better? More clear? There is no right answer to my question by the way, only good and less useful ones. :) $\endgroup$
    – Charlie Parker
    Apr 8, 2014 at 23:31
  • 2
    $\begingroup$ ecdf = empirical cdf, sorry. We can only answer the question you ask, not the one you meant to ask, so you have to be careful to be clear when you express your assumptions. $\endgroup$
    – Glen_b
    Apr 8, 2014 at 23:32
  • 1
    $\begingroup$ A normalized histogram can be seen as a density estimate $\endgroup$
    – Dason
    Apr 9, 2014 at 0:47

3 Answers 3

Reset to default
4
$\begingroup$

You don't specify that you're talking about continuous random variables, but I'll assume, since you mention KDE, that you intend this.

Two other methods for fitting smooth densities:

1) log-spline density estimation. Here a spline curve is fitted to the log-density.

An example paper:

Kooperberg and Stone (1991),
"A study of logspline density estimation,"
Computational Statistics & Data Analysis, 12, 327-347

Kooperberg provides a link to a pdf of his paper here, under "1991".

If you use R, there's a package for this. An example of a fit generated by it is here. Below is a histogram of the logs of the data set there, and reproductions of the logspline and kernel density estimates from the answer:

histogram of log-data

$$ $$

Logspline density estimate:

logspline plot

Kernel density estimate:

kernel density estimate

2) Finite mixture models. Here some convenient family of distributions is chosen (in many cases, the normal), and the density is assumed to be a mixture of several different members of that family. Note that kernel density estimates can be seen as such a mixture (with a Gaussian kernel, they're a mixture of Gaussians).

More generally these might be fitted via ML, or the EM algorithm, or in some cases via moment matching, though in particular circumstances other approaches may be feasible.

(There are a plethora of R packages that do various forms of mixture modelling.)

Added in edit:

3) Averaged shifted histograms
(which are not literally smooth, but perhaps smooth enough for your unstated criteria):

Imagine computing a sequence of histograms at some fixed binwidth ($b$), across a bin-origin that shifts by $b/k$ for some integer $k$ each time, and then averaged. This looks at first glance like a histogram done at binwidth $b/k$, but is much smoother.

E.g. compute 4 histograms each at binwidth 1, but offset by +0,+0.25,+0.5,+0.75 and then average the heights at any given $x$. You end up with something like so:

Averaged shifted histogram

Diagram taken from this answer. As I say there, if you go to that level of effort, you might as well do kernel density estimation.

Improve this answer
$\endgroup$
1
  • $\begingroup$ To add to this. For the mixture model - I guess you could fit a mixture of 2, then 3, then 4 distributions and stop after there is no significant increase in log-likelihood or some such... $\endgroup$
    – waferthin
    Apr 17, 2014 at 14:26
4
+25
$\begingroup$

Subject to the comments above about assumptions such as smoothness etc. You can do Bayesian nonparametric density estimation using mixture models with the Dirichlet process prior.

The picture below shows the probability density contours recovered from MCMC estimation of a bivariate normal DP-mixture model for the 'old faithful' data. The points are coloured IIRC according to the clustering obtained on the last MCMC step.

enter image description here

Teh 2010 provides some good background.

Improve this answer
$\endgroup$
1
$\begingroup$

A popular choice are random forest (see concretely chapter five of "Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning".

It describes in detail the algorithm and evaluates it against other popular choices like k-means, GMM and KDE. Random Forest are implemented in R and scikit-learn.

Random Forest are bagged decision trees in a clever way.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.