Distance between empirically generated distributions (in R)

Question

I'm not a statistician, but I sometimes need to play around with data. I have two data sets, lists of values in the unit interval. I've plotted them as histograms, so I have an intuitive idea of how "far apart" they are. But I want to do something a little more formal.

My first thought was to just sum the differences of the values in the bins, but this isn't that satisfactory. Then I thought of taking a three-bin average and sum differences over these. (Apologies if I'm mangling statistics terminology)

But I was thinking I'm probably reinventing the wheel, so I came here. Similar questions seem to point to "Kolmogorov Smirnov tests" or something like that.

So my question is: is this the right method to calculate how far these data sets are apart? And is there an easy way to do this in R? Ideally just KStest(data1,data2) or something?

Edit To emphasize, I'm particularly interested in ways to measure how far the data are apart directly rather than fitting a distribution to each and then measuring the distance between distributions. [Does that even make sense? I guess numerical calculations in R will be done by sampling from a distribution anyway.]

Answer 1

You can do a Kolmogorov-Smirnov test using the ks.test function. See ?ks.test.

In general, when you are looking for a function in R (and you don't know its name) try using ??. For instance, ??"Kolmogorov Smirnov". If nothing comes up RSiteSearch("whatever you're looking for") should help :)

Answer 2

A standard way to compare distributions is to use the Kullback-Leibler divergence. As usual, there's an R package that does this for you! From the ?KLdiv help page in the flexmix package, we get the following bit of code:

## Gaussian and Student t are much closer to each other than
## to the uniform:
> library(flexmix)
> x = seq(-3, 3, length=200)
> y = cbind(u=dunif(x), n=dnorm(x), t=dt(x, df=10))
> matplot(x, y, type="l")

> round(KLdiv(y),3)
      u     n     t
u 0.000 1.082 1.108
n 4.661 0.000 0.004
t 4.686 0.005 0.000

Notice that the comparison isn't symmetric: so uniform vs Normal is different from Normal vs Uniform.

You didn't explain why you wanted to compare distributions. Giving a use-case may get you more specific answers.

Answer 3

First thing: define "distance". That sounds like a stupid question, but what do you mean by distance? Is the data paired? Then -and only then- it makes sense to look at the sum of (squared) differences to decide about the distance between two datasets. If not, you have to resort to other means.

The next question is: is the data distributed in the same manner? If so, you can see the difference between the means as the "location shift" of your data (or the distance between both datasets).

But if neither of both is true, how do you define the distance between datasets then? Do you take shape of the distribution into account for example? You really have to think about those issues before trying to calculate a distance.

This said : One (naive) possibility is to use the mean of the differences between all possible x-y combinations. Formalized this is :

$$Dist=\sqrt{\frac{1}{n_1 n_2}\sum_{i=1}^{n_1} \sum_{j=1}^{n_2}(X_i - Y_j)^2}$$

In R :

x <- rnorm(10)
y <- rnorm(10,2)
sqrt(mean(outer(x,y,"-")^2))

If you allow for negative distances, you can drop the sqrt and the ^2 :

mean(outer(x,y,"-"))

A simulation shows easily that this will give indeed the difference between the means in the example, as both distributions are equal in this case. But be warned that negative distances are not allowed in many applications. In the first scenario, the number will always be a bit larger than the difference between the mean. In any case, if you're interested in the difference between the center of your datasets, define the center and calculate the difference between those centers. That might very well be what you're after.

Contrary to the other suggestions, this approach does not make any assumptions about the distribution of your data. Which makes it applicable in all situations, but also difficult to interpret.