Rolling sum of 2 sample KS test results

Question

In order to compare two lists of samples from 'before-treatment' and 'after-treatment', I am doing a two sample KS test using the ks_2samp function from Python's scipy.stats package which gives me the D and p-value statistics. Now I want to do this on a rolling basis. I periodically receive new samples and I want to find a way to reuse the KS statistics calculated from the previous set of samples.

E.g. if one set of before-treatment samples are in x and after-treatment in y then:

x = [1,1,1,1]

y = [2,2,2,1,1,1]

D, p_value = ks_2samp(x, y)

Output is:

D = 0.5, p_value = 0.43531018975534286

In next set of samples:

x = [1,1,1,1,1,1,1,1]

y = [2,2,2,2,2,1,1,1,1,1]

D, p_value = ks_2samp(x, y)

Output is:

D = 0.5, p_value = 0.14848228822491449

Note that for a given x and y, the number of 'before' samples in x and 'after' samples in y can differ. Also note that, the samples in x and y are from independent set of experiments.

Now I want to somehow combine the D and p_values from both of these separate ks test runs and get some idea of the overall result. Should I just take the mean or is there some other way that makes more sense? I don't have much knowledge of statistics so apologies if this is a very trivial question.

Answer 1

So this one does not go unanswered I repeat here material from my comments. It is perfectly possible to combine the $p$-values using any of the standard methods. For the two values quoted in the question using four of the most common methods we get.

eponym p Tippett 0.27478 Fisher 0.24157 Edgington 0.17035 Stouffer 0.19685

I know of no way of combining Kolmogorov Smirnov statistics and I do not see what scientific question that would answer.

Answer 2

The KS test only has a limited number of possible p values, in particular for small sample sizes. Standard methods for combining p values, however, are based on the assumption that p values are uniformly distributed on (0,1] under the null hypothesis. Applying them to results of the KS test with a small sample size will usually considerably overestimate the p value.

I wrote a Python package that addresses this problem, however, like other answerers and commentors I see no reasonable application to combine p values of the KS test, which is why it is not straightforwardly supported and we need to do a bit of work yourself determining all possible p values under the null hypothesis.

The following Python script combines your p values:

import numpy as np
from scipy.stats import ks_2samp
from combine_pvalues_discrete import CTR, combine

x1 = [1,1,1,1]
y1 = [2,2,2,1,1,1]
x2 = [1,1,1,1,1,1,1,1]
y2 = [2,2,2,2,2,1,1,1,1,1]

def ones_and_twos(template,number_of_twos):
    result = np.ones_like(template)
    result[:number_of_twos] = 2
    return result

def ks_ctr(x,y):
    total_twos = sum(np.hstack((x,y))==2)
    p = ks_2samp(x,y).pvalue
    
    possible_ps = [
            ks_2samp(
                    ones_and_twos( x, twos_in_x ),
                    ones_and_twos( y, total_twos-twos_in_x ),
                ).pvalue
            for twos_in_x in range(total_twos+1)
        ]
    
    return CTR(p,possible_ps)

ctrs = [ ks_ctr(x,y) for x,y in [(x1,y1),(x2,y2)] ]

for method in ["fisher","mudholkar_george","edgington","stouffer","tippett"]:
    print( method, combine(ctrs,method=method) )

You get:

• Fisher: 0.17
• Mudholkar–George: 0.13
• Edgington: 0.11
• Stouffer: 0.11
• Tippett 0.25