Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a well accepted measure between two continuous densities, but it's not clear how to apply that to a set of samples.


For visualization purposes, try a Q-Q plot, which is a plot of the quantiles of your data against the quantiles of the expected distribution.

If you want a statistical test, the Kolmogorov-Smirnov statistic provides a non-parametric test for whether the data come from $p(x)$, using the maximum difference in the empirical and analytic cdf.

Of course, you could also evaluate the log-probability of your data under the two distributions: $L_1 = \sum_i p_1(X_i)$ vs. $L_2 = \sum_i p_2(X_i)$, and take whichever is larger. This is equivalent to maximum likelihood density comparison. (However, this may not be valid if $p_1$ and $p_2$ are distributions fit to your data, especially if they have different numbers of fitted parameters; in that case you want to do "model comparison", and there are a variety of tools for this— AIC, BIC, Bayes Factors, Likelihood-ratio test, Cross-validation, etc.)

| cite | improve this answer | |
  • $\begingroup$ +1 great answer, but it looks like I need the cdf of p(x) to compute the Kolmogorov-Smirnov statistic. Unfortunately I don't have that, in fact I only have p(x) up to a normalizing constant (which is why I'm approximating it with samples). $\endgroup$ – fairidox Jul 8 '11 at 8:04
  • $\begingroup$ Also, I have a question about log likelihood. Imagine I have a complex multi-modal function, and I have a set of samples all focused around one high density point. This is clearly a poor approximation, but it would have a large likelihood, in fact the maximum likelihood set of points is N points all exactly at the global max. $\endgroup$ – fairidox Jul 8 '11 at 8:25
  • $\begingroup$ Ah, ok—I understand your question a bit better. You're right that N points at the mode will have highest likelihood (making the likelihood a poor statistic to use if your goal is to get a "typical" sample from $p(x)$). I should have said that Q-Q and K-S both (typically) get applied to 1D distributions. Is that the case here? $\endgroup$ – jpillow Jul 8 '11 at 9:03
  • 1
    $\begingroup$ Final thought: if your density is 1D (or even low-D) and you can evaluate it up to a constant of proportionality, you can still construct the CDF by evaluating $p(x)$ at evenly-spaced increments and normalizing by the sum of all values $p(x_i)$ (i.e., so that the cdf goes from 0 to 1). $\endgroup$ – jpillow Jul 8 '11 at 9:07
  • 1
    $\begingroup$ Accepted the answer, I think the KS-statistic is what I need after all. Ultimately I am looking to test the quality of a particular sampling technique, for this analysis I can stick to functions where the the cdf is known, and estimating it online is not really required. $\endgroup$ – fairidox Jul 8 '11 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.