Similarity between closed-form and sampling-based distribution Are there any well-known techniques that are capable of estimating the similarity between a closed-form distribution (specifically a Gaussian if that helps) and a distribution obtained by sampling? I'm not sure what the metrics are typically used for similarity estimation but one that makes sense intuitively would be to just find some measure of overlap between the two distributions.
Would I be better off using a smooth approximation of the discrete sampling-based distribution (not sure if this is something that is straightforward to do in the multi-dimensional case) available or are there methods to work directly with such a discrete distribution?
 A: You can follow the idea of statistical tests for a special distributional shape and base the distance measure on the distance between the empirical cumulative distribution function $F_{emp}$ and the theoretical distribution function $F$. Then you can use the supremum distance
$$D_{sup}=\sup_{x}|F_{emp}(x) - F(x)|$$
or the $L_p$ distance
$$D_p=\left(\int_{-\infty}^{\infty} |F_{emp}(x) - F(x)|^p dx\right)^{1/p}$$
As $F_{emp}$ is a step function, both the supremum and the intergral can be computed from the $n$ sample values $x_1,\ldots,x_n$.
The supremum distance is used in the Kolmogorov-Smirnov Test as a test statistic, and the (square of the) $L_2$ distance by the Crame-van-Mieses test, which means that even their (approximate) distributions are known if the data are indeed generated by the comparison distribution. This allows for computing p-values for $P(D_{sup}>d)$ or $P(D_2^2>d)$, where $d$ is the measured value. One minus the p-value can also be considered a distance measure, although I am not sure whether it is zero for an exact match (if not, it would not qualify as a distance measure).
A: You did not say what exactly you mean by "a distribution obtained by sampling". Besides an ecdf you could also use just histograms of some resolution or KDE or any of many other methods. Knowing how you do this might help in deciding how to answer your question.
In general, there are many notions of "distance" between distributions, e.g. various divergences like the KL-divergence, or even proper distances like e.g. Wasserstein metric.
As far as your question about the intuition of density similarity and "overlap" is concerned, you might want to have a look at the Wasserstein metric. It is also called  "earth mover's distance" (EMD) because of its very intuitive nature. Just imagine the two densities, the one you obtained from the samples, and the one you want to compare it to, as two heaps of earth. The EMD is then the minimal amount of dirt to move to change the first heap into the second.
