# How do I determine how well a dataset approximates a distribution?

Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a well accepted measure between two continuous densities, but it's not clear how to apply that to a set of samples.

If you want a statistical test, the Kolmogorov-Smirnov statistic provides a non-parametric test for whether the data come from $p(x)$, using the maximum difference in the empirical and analytic cdf.
Of course, you could also evaluate the log-probability of your data under the two distributions: $L_1 = \sum_i p_1(X_i)$ vs. $L_2 = \sum_i p_2(X_i)$, and take whichever is larger. This is equivalent to maximum likelihood density comparison. (However, this may not be valid if $p_1$ and $p_2$ are distributions fit to your data, especially if they have different numbers of fitted parameters; in that case you want to do "model comparison", and there are a variety of tools for this— AIC, BIC, Bayes Factors, Likelihood-ratio test, Cross-validation, etc.)
• Ah, ok—I understand your question a bit better. You're right that N points at the mode will have highest likelihood (making the likelihood a poor statistic to use if your goal is to get a "typical" sample from $p(x)$). I should have said that Q-Q and K-S both (typically) get applied to 1D distributions. Is that the case here? – jpillow Jul 8 '11 at 9:03
• Final thought: if your density is 1D (or even low-D) and you can evaluate it up to a constant of proportionality, you can still construct the CDF by evaluating $p(x)$ at evenly-spaced increments and normalizing by the sum of all values $p(x_i)$ (i.e., so that the cdf goes from 0 to 1). – jpillow Jul 8 '11 at 9:07