Using KS test to evaluate differences

There are two different empirical CDFs which I have and I would like to evaluate how different they are. The place at which I started was Kolmogorov-Smirnov test - the result in my case looks like this (from the R ks.test)

Two-sample Kolmogorov-Smirnov test

data:  wr and j2
D = 0.2036, p-value = 1.965e-14
alternative hypothesis: two-sided

while it is okay to be able to say that they might not be from the same underlying distribution I was wondering whether the D reported porvides me with a measure of how dissimilar they are? Can I say that two distributions with D 0.4 are more dissimilar than with 0.2..?

The Kolmogorov–Smirnov test statistic $D$ estimates the maximum absolute difference in the distribution functions of the two populations from which the two samples are drawn. There are obviously many different ways you could quantify the dissimilarity between two distributions but this is one, & using $D$ as a test statistic already rather implies you're happy with it. As @stochazesthal says, it's always a good idea to plot the empirical distribution functions & see how they're dissimilar.

Since the test statistics of Kolmogorov-Smirnov is defined as $$D = sup_{x}(F(x) - G(x))$$

you can't say that two distributions with $D = 0.4$ are more dissimilar than two distributions with $D = 0.2$. Indeed, you are accounting only for the maximal element of a vector of differences.

However, I think you could compare the distributions of all the values of $D_{all} = F(x) - G(x)$. Such a comparison should give you additional insights to better understand how and where two couple of distributions are different.

Instead, if you are looking for multiple-samples versione of the Kolmogorov-Smirnov, you can take a look at this: Is there a multiple-sample version or alternative to the Kolmogorov-Smirnov Test?

Not exactly since you must keep track of which quantiles have caused the largest difference(s) in CDFs, say there were a few quantiles, it then becomes ambiguous.

In my opinion, a better way to quantify the difference would be to use Kullback-Leibler divergence from library(flexmix); KLdiv(varmx). However, this measure is not symmetric and it uses raw data, though this might be good, if you have a few candidate empirical distribution realizations to evaluate, with respect to a given distribution (be it empirical or theoretical).

For usage of the K-L, you can consult this thread

• but for KL - dont you need the functional form of both cdfs..dont you? so i think otherwise KL would be better but I do not have the precise form jus the numerical vector with they y values fo the cdfs... Commented May 23, 2015 at 12:01
• You can compute KL divergence starting from the two numerical vectors :) Commented May 23, 2015 at 12:04
• @stochazesthai what do you mean precisely...in RE (R) the input is dist1, dist2 other paramters so I dont think I could just input the two numerical vectors... Commented May 23, 2015 at 12:11
• you mean get the ecdf from the numerical vectors and then compute the kubach leibkler div? Commented May 23, 2015 at 12:18
• For the K-L, you need the empirical density function, which can be estimated from the raw data with the $density$ function in R. I have added a reference for K-L to my answer. You can check it (it has examples with Normals) Commented May 23, 2015 at 14:24