Bootstrapping and Kolmogorov-Smirnov Here is an experiment I did:


*

*I bootstrapped a sample $S$ and stored the results as empirical distribution under the name $S_1$.

*Then I bootstrapped $i=10000$ times in a row the same sample $S$ and compare the resulting empirical distributions $S_i$ with $S_1$ using Kolmogorov-Smirnov test .


Results from the experiment: The comparisons return different $p$-values (from $0.01$ to $0.99$) and different $D$ values (from $0.02$ to $0.06$).
Is that expected? If I bootstrap the same sample 1000 times isn't it expected that all 1000 empirical distributions to be from the same distribution?
If yes then should I try to establish the distribution of the empirical distributions ($S_1$, $S_i$)?
For instance:
Three empirical distributions $S_1$, $S_2$, $S_3$ bootstrapped from the same initial sample $S$:
S1: 1,2,3,4,5,6
S2: 1,3,4,5,6,7
S3: 2,4,5,6,7,8

If I add them up I get:
1,1,2,2,3,3,4,4,4,5,5,5,6,6,6,7,7,8

 A: The thing to recognize here is that all of your bootsamples come from the same population.  That is, the null hypothesis obtains here.  Bear in mind that under the null hypothesis, the $p$-value is distributed as a uniform.  So it sounds like everything worked fine (although I don't know if that is what you were trying to do).  
A: I think I understand your problem now. You alluded to your assumption that somehow KS test should show that all bootstrapped samples should be shown to be from the original sample. However, consider this: what does it mean to show that they're from the same distribution?
It usually means that p-value is over some $\alpha$ confidence. If bootstrapping is done properly you'll get p-value sometimes over, sometimes under the $\alpha$. Build the distribution of test statistics you get from running KS test on bootstrapped samples. Observe p-values for various critical values, they should match the theoretical values for which KS test was designed.
A: It seems to me that you could apply permutation testing for checking whether your samples come from the same distribution. It is similar to Boostrapping in the sense that you will make a Monte-Carlo simulation.
Under the Null hypothesis you can permute all the observations from one set to another. Then you (repeatedly) compute the statistic of interest on the permuted sets. This is a really nice website explaining the logic behind permuation tests: https://www.jwilber.me/permutationtest/
