4
$\begingroup$

Assume that for a univariate sample $X$ you have a density estimate by a model you developed (in my case it's a density forecasting model) and you want to assess the quality of your models estimate.

Using artificial data generated from a particular distribution, e.g. a mixture distribution, you can directly compare the models estimates to the pdf of the underlying distribution, given that both the estimate and the true pdf are evaluated at the same set of points. Using common error or divergence measures, we can get a measure for the quality of the estimate.

But how to proceed, if you are working with real data of which you don't know the underlying distribution and therefore can only work with the actual instances in the sample?

Consider the example below, for simplicity's sake I took some artificial data from a mixture of skew-normal distributions:
I've visualized a histogram of the sample (relative instead of absolute frequencies), a spline interpolation of the histogram, the true underlying density as well as two kernel density estimators, one with silverman's rule of thumb (label $\textrm{KDE}_{rot}$) and a handpicked bandwidth (30/N).
The top graphic shows the plots for a random subsample of the actual data and the bottom plot shows the full sample, since in my application I will need to work with varying sample sizes too.

It can be seen that while the underlying distribution is the same, of course the differ due to the different sample sizes. The problem with the KDEs is determining a fitting bandwidth.

Is using a smoothed histogram really the best option in this case? Are there alternatives that I missed?

enter image description hereenter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The most straightforward estimate of a distribution from which a random sample comes is the empirical distribution. The empirical distribution differs from a histogram or KDE in that it isn't smoothed or grouped. So you might compare your proposed distribution to the empirical distribution; this provides a measure of how well your proposed distribution fits the data. Of course, overfitting can be an issue, for which you should consider the usual fixes like cross-validation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I wanted to get back to this answer but completely forgot about it. I thought of this too, but isn't the empirical distribution a continuous one, which therefore has a probability mass function rather than a probability density function (see this question) ? Can I still arrive at an approximation that I can compare to my estimate of the pdf? $\endgroup$
    – deemel
    Commented Jul 24, 2018 at 15:26
  • $\begingroup$ @Rickyfox Yes and yes. You can integrate your PDF to get a CDF, and all random variables have CDFs, so you can compare that CDF to the empirical CDF with e.g. a quantile-quantile plot. Or you can interpolate the empirical distribution into a continuous distribution; for example, you can connect each adjacent pair of points in the empirical PMF with a line segment, then divide everything by the integral to normalize it to have an integral of 1. $\endgroup$
    – Kodiologist
    Commented Jul 24, 2018 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.