Measuring distance between two empirical distributions This is similar to the question  Measuring the "distance" between two multivariate distributions, except that I want to measure distance between two data sets. I can imagine simply computing the mean, standard deviation, and other summary statistics and then somehow aggregating the differences for each distribution, but that does not seem very theoretically sound.
I wonder if there is a generally accepted method for doing this measurement.
 A: I think your question is essentially the same as Can I use Kolmogorov-Smirnov to compare two empirical distributions?, for which the Kolmogorov-Smirnov test is commonly used. The KS test statistic is a natural measure of the distance between two samples. For more details, please see Kolmogorov–Smirnov test on Wikipedia.
A: An area in statistics where this problem arises naturally is Approximate Bayesian Computation.
What you actually want to do is to summarise the whole sample into "informative" statistics that can later be compared using a suitable metric: this problem is not trivial at all. I would even say that it is actually one of the "hot topics" in statistics.
It is not that the method is not theoretical, it is just that there is not unique way of comparing two data sets. It usually depends on your aims and your model.
If you have a model and can identify sufficient statistics for it, then, by comparing the sufficient statistics of both samples you can assess how different the information contained in each sample is. If they are very close, then the associated [likelihood functions] (http://en.wikipedia.org/wiki/Likelihood_function) would be similar, and therefore the inferences on the corresponding parameters would be similar as well.
