Bootstrapping and Kolmogorov-Smirnov

Question

Here is an experiment I did:

1. I bootstrapped a sample $S$ and stored the results as empirical distribution under the name $S_1$.
2. Then I bootstrapped $i=10000$ times in a row the same sample $S$ and compare the resulting empirical distributions $S_i$ with $S_1$ using Kolmogorov-Smirnov test .

Results from the experiment: The comparisons return different $p$-values (from $0.01$ to $0.99$) and different $D$ values (from $0.02$ to $0.06$).

Is that expected? If I bootstrap the same sample 1000 times isn't it expected that all 1000 empirical distributions to be from the same distribution?

If yes then should I try to establish the distribution of the empirical distributions ($S_1$, $S_i$)?

For instance: Three empirical distributions $S_1$, $S_2$, $S_3$ bootstrapped from the same initial sample $S$:

S1: 1,2,3,4,5,6
S2: 1,3,4,5,6,7
S3: 2,4,5,6,7,8

If I add them up I get:

1,1,2,2,3,3,4,4,4,5,5,5,6,6,6,7,7,8

Answer 1

I think I understand your problem now. You alluded to your assumption that somehow KS test should show that all bootstrapped samples should be shown to be from the original sample. However, consider this: what does it mean to show that they're from the same distribution?

It usually means that p-value is over some $\alpha$ confidence. If bootstrapping is done properly you'll get p-value sometimes over, sometimes under the $\alpha$. Build the distribution of test statistics you get from running KS test on bootstrapped samples. Observe p-values for various critical values, they should match the theoretical values for which KS test was designed.

Answer 2

The thing to recognize here is that all of your bootsamples come from the same population. That is, the null hypothesis obtains here. Bear in mind that under the null hypothesis, the $p$-value is distributed as a uniform. So it sounds like everything worked fine (although I don't know if that is what you were trying to do).

Answer 3

It seems to me that you could apply permutation testing for checking whether your samples come from the same distribution. It is similar to Boostrapping in the sense that you will make a Monte-Carlo simulation. Under the Null hypothesis you can permute all the observations from one set to another. Then you (repeatedly) compute the statistic of interest on the permuted sets. This is a really nice website explaining the logic behind permuation tests: https://www.jwilber.me/permutationtest/

Answer 4

Yes, it is expected.

Bootstrapping simulates sampling multiple samples (where "sample" here stands for a collection of data points, not a single datum) from the same, underlying, parent distribution. Therefore, sometimes you will get something that is very close to $S_1$, sometimes not, stochastically. The point is how often you get something significantly different from $S_1$: you just mention the range $0.01$ to $0.99$, not the shape of the distribution of these results, and those figures are not worrisome per se (see Aksakal's answer here).

PS: As for any statistical test, you can only reject the null hypothesis, not confirm it. Therefore, strictly speaking, you cannot state: "[..] KS-test does not show that all empirical distributions come from the same underlying distribution", although I understand what you meant, in this context.