Do Dvoretzky–Kiefer–Wolfowitz bounds and Kolmogorov-Smirnov test different hypotheses?

Question

This question is related to my previous question that was answered excellently by Matt Krause.

I am trying to compare two data vectors to find out if they come from the same distribution. I computed both K-S test results as well as DKW bound. I uses kstest2 function of MATLAB (with default alpha value of 0.05). This gives back the KS2 Statistic (ks2stat, in short), as well as the decision. In K-S test, the null hypothesis is the two data vectors come from the same distribution; alternative being they do not.

Next, I computed the DKW bound from the ECDF of each data vector with alpha = 0.05. Let's call these DKW1 and DKW2 respectively. As explained here (Section D), I should add the DKW bounds of both ECDFs and then compare it with the K-S distance (which is ks2stat as obtained in the first step).

DKW1 = sqrt(ln(2/alpha)/(2*length(datavector1))); 
DKW2 = sqrt(ln(2/alpha)/(2*length(datavector2))); 
DKW = DKW1 + DKW2;

Then, as per the previous link, if ks2stat > DKW, then the two data vectors do NOT come from the same distribution. Otherwise, either the two data vectors do come from the same distribution or the number of samples are not large enough to make any decision.

I noticed that there were quite a few cases when KS2 test gave H1 (alternative) hypothesis as the decision while ks2stat turned out to be less than DKW. Does this mean KS2 test and DKW-based test are testing different hypotheses? Could someone please explain? Any help would be greatly appreciated. Thanks.

Regards, RD

Answer 1

The standard DKW bounds are an inversion of the 1-sample KS test and thus test exactly the same thing. However, ks2stat is a two-sample version version which is a bit different. Adding the DKW bounds together is an overly conservative approach to the 2-sample DKW bounds. The paper you cite uses the triangle inequality to ensure that summing them together is at least as large as needed. The following article discusses the relationship and inversion of the KS test in the 2-sample case and describes when it holds. The TL;DR is that when you have samples of length n and m, you want to scale the bounds by

$$ \sqrt{\frac{n + m}{n m}} $$

instead of the normal KS / DKW formulas which are scaled by

$$ \sqrt{\frac{1}{n}} $$

https://arxiv.org/pdf/1107.5356.pdf