Using KS test to evaluate differences

Question

There are two different empirical CDFs which I have and I would like to evaluate how different they are. The place at which I started was Kolmogorov-Smirnov test - the result in my case looks like this (from the R ks.test)

Two-sample Kolmogorov-Smirnov test

data:  wr and j2
D = 0.2036, p-value = 1.965e-14
alternative hypothesis: two-sided

while it is okay to be able to say that they might not be from the same underlying distribution I was wondering whether the D reported porvides me with a measure of how dissimilar they are? Can I say that two distributions with D 0.4 are more dissimilar than with 0.2..?

Answer 1

The Kolmogorov–Smirnov test statistic $D$ estimates the maximum absolute difference in the distribution functions of the two populations from which the two samples are drawn. There are obviously many different ways you could quantify the dissimilarity between two distributions but this is one, & using $D$ as a test statistic already rather implies you're happy with it. As @stochazesthal says, it's always a good idea to plot the empirical distribution functions & see how they're dissimilar.

Answer 2

Since the test statistics of Kolmogorov-Smirnov is defined as $$ D = sup_{x}(F(x) - G(x))$$

you can't say that two distributions with $D = 0.4$ are more dissimilar than two distributions with $D = 0.2$. Indeed, you are accounting only for the maximal element of a vector of differences.

However, I think you could compare the distributions of all the values of $D_{all} = F(x) - G(x)$. Such a comparison should give you additional insights to better understand how and where two couple of distributions are different.

Instead, if you are looking for multiple-samples versione of the Kolmogorov-Smirnov, you can take a look at this: Is there a multiple-sample version or alternative to the Kolmogorov-Smirnov Test?

Answer 3

Not exactly since you must keep track of which quantiles have caused the largest difference(s) in CDFs, say there were a few quantiles, it then becomes ambiguous.

In my opinion, a better way to quantify the difference would be to use Kullback-Leibler divergence from library(flexmix); KLdiv(varmx). However, this measure is not symmetric and it uses raw data, though this might be good, if you have a few candidate empirical distribution realizations to evaluate, with respect to a given distribution (be it empirical or theoretical).

For usage of the K-L, you can consult this thread