How to estimate required sample size for the two-sample Kolmogorov-Smirnov test

Question

I have a simulation model which produces a value for an output variable. Running it many times gives skewed distributions for this output variable. What I'd like to do is compare two distributions, corresponding to the results arising from two different sets of input parameters to the simulation model.

If I have run the simulation model, say for 10 replications each, is there some way I can use the observed sample means and variances of the two sample populations to say how many replications are likely to be needed to reject the null hypothesis that the populations are the same with a certain confidence level? If I understand correctly the two-sample Kolmogorov-Smirnov test can be used to test the null hypothesis but I'm looking for a way to understand whether the number of samples I have already is enough to determine this with a certain level of confidence.

In the simplest terms I'd really like to be able to eyeball some boxplots, make a judgement over which set of input parameters to the simulation model are best, and understand whether the number of simulation replications is enough to be confident in the conclusion.

Answer 1

There are bounds that can be used for estimating the distribution of the maximum absolute error in an empirical cumulative distribution function. One calculation I made is that you need about 190 observations to estimate one ECDF. To estimate the difference in two ECDFs like the Kolmogorov test does probably takes 4x as many observations. Estimation is different from your hypothesis testing question but does give a ballpark estimate.