Can I use Kolmogorov-Smirnov to compare two empirical distributions?

Question

Is it OK to use the Kolmogorov-Smirnov goodness-of-fit test to compare two empirical distributions to determine whether they appear to have come from the same underlying distribution, rather than to compare one empirical distribution to a pre-specified reference distribution?

Let me try asking this another way. I collect N samples from some distribution at one location. I collect M samples at another location. The data is continuous (each sample is a real number between 0 and 10, say) but not normally distributed. I want to test whether these N+M samples all come from the same underlying distribution. Is it reasonable to use the Kolmogorov-Smirnov test for this purpose?

In particular, I could compute the empirical distribution $F_0$ from the $N$ samples, and the empirical distribution $F_1$ from the $M$ samples. Then, I could compute the Kolmogorov-Smirnov test statistic to measure the distance between $F_0$ and $F_1$: i.e., compute $D = \sup_x |F_0(x) - F_1(x)|$, and use $D$ as my test statistic as in the Kolmogorov-Smirnov test for goodness of fit. Is this a reasonable approach?

(I read elsewhere that the Kolmogorov-Smirnov test for goodness of fit is not valid for discrete distributions, but I admit I don't understand what this means or why it might be true. Does that means my proposed approach is a bad one?)

Or, do you recommend something else instead?

Answer 1

That is OK, and quite reasonable. It is referred to as the two-sample Kolmogorov-Smirnov test. Measuring the difference between two distribution functions by the supnorm is always sensible, but to do a formal test you want to know the distribution under the hypothesis that the two samples are independent and each i.i.d. from the same underlying distribution. To rely on the usual asymptotic theory you will need continuity of the underlying common distribution (not of the empirical distributions). See the Wikipedia page linked to above for more details.

In R, you can use the ks.test, which computes exact $p$-values for small sample sizes.