Bootstrapping and Kolmogorov-Smirnov

Here is an experiment I did:

1. I bootstrapped a sample $S$ and stored the results as empirical distribution under the name $S_1$.
2. Then I bootstrapped $i=10000$ times in a row the same sample $S$ and compare the resulting empirical distributions $S_i$ with $S_1$ using Kolmogorov-Smirnov test .

Results from the experiment: The comparisons return different $p$-values (from $0.01$ to $0.99$) and different $D$ values (from $0.02$ to $0.06$).

Is that expected? If I bootstrap the same sample 1000 times isn't it expected that all 1000 empirical distributions to be from the same distribution?

If yes then should I try to establish the distribution of the empirical distributions ($S_1$, $S_i$)?

For instance: Three empirical distributions $S_1$, $S_2$, $S_3$ bootstrapped from the same initial sample $S$:

S1: 1,2,3,4,5,6
S2: 1,3,4,5,6,7
S3: 2,4,5,6,7,8


If I add them up I get:

1,1,2,2,3,3,4,4,4,5,5,5,6,6,6,7,7,8

• What's the objective? Did you look at Lilliefors test and the paper which describes it? Dec 10, 2014 at 13:56
• The objective is to be able to KS-test empirical distributions established bootstrapped from different samples. Hence I test KS-test on the empirical distributions bootstrapped from one and the same underlying sample. As of now KS-test does not show that all empirical distributions come from the same underlying distribution which is strange. Dec 10, 2014 at 14:15
• That's the point of bootstrapping. They are not supposed to. Look at test stats and compare with critical values. Dec 10, 2014 at 14:17
• What do you mean by "they are no supposed to"? If I bootstrap the same sample 1000 times isn't it expected that all 1000 empirical distributions to be from the same distribution? Dec 10, 2014 at 15:55
• It depends on how you bootstrap. That's why I asked if you looked at how it's done properly, like in Lilliefors' paper, for instance. Dec 10, 2014 at 16:09

I think I understand your problem now. You alluded to your assumption that somehow KS test should show that all bootstrapped samples should be shown to be from the original sample. However, consider this: what does it mean to show that they're from the same distribution?

It usually means that p-value is over some $\alpha$ confidence. If bootstrapping is done properly you'll get p-value sometimes over, sometimes under the $\alpha$. Build the distribution of test statistics you get from running KS test on bootstrapped samples. Observe p-values for various critical values, they should match the theoretical values for which KS test was designed.

• Yes my expectation is that KS test should show that all bootstrapped samples should be shown to be from the original sample. Note that the bootstrapped samples I KS-test are of the same size (1000). If I build the distribution of KS-test statistics how this will help me when I start comparing if bootstrapped samples from two different samples are actually drawn from the same underlying distribution? Do you have a paper to recommend? Dec 10, 2014 at 16:32
• This is the paper I referred to. It's about KS-test. Also see the discussion here with code example: stats.stackexchange.com/questions/126539/… Dec 10, 2014 at 16:36

The thing to recognize here is that all of your bootsamples come from the same population. That is, the null hypothesis obtains here. Bear in mind that under the null hypothesis, the $p$-value is distributed as a uniform. So it sounds like everything worked fine (although I don't know if that is what you were trying to do).

• My objective is using KS-test to compare if bootstrapped empirical distributions coming from different populations are actually drawn from the same underlying distribution. Hence the first thing I tried was to test that with one population and it looks like null hypothesis doesn't hold. Dec 10, 2014 at 16:09
• I don't see why you think the null doesn't hold. You state that the p-values are distributed from .01 to .99. That's what they are supposed to do under the null. Dec 10, 2014 at 16:12
• Isn't it that p-value should be >0.1 to have no presumption against the null hypothesis? Dec 10, 2014 at 16:14
• No. Where did you get "0.1"? This is your confidence level, which you set. What if I want 0.05? How would your bootstrapping know that I want 0.05? You'll be getting different p-values every time. Dec 10, 2014 at 16:20
• @DimitarBakardzhiev, Aksakal is right. The p-value is not 'supposed' to be >0.1 when the null holds. It is supposed to take all values (0, 1.0) equally often. Dec 10, 2014 at 16:22

It seems to me that you could apply permutation testing for checking whether your samples come from the same distribution. It is similar to Boostrapping in the sense that you will make a Monte-Carlo simulation. Under the Null hypothesis you can permute all the observations from one set to another. Then you (repeatedly) compute the statistic of interest on the permuted sets. This is a really nice website explaining the logic behind permuation tests: https://www.jwilber.me/permutationtest/