# Test and minimum sample size to conclude that 2 ECDFs are different

Let's say I have samples $$X$$ from one distribution and I am now collecting new samples $$Y$$ from another unknown distribution. What would be the best test to check whether the samples $$Y$$ come from the same distribution of $$X$$? Should I use the Kolmogorov-Smirnov test? And how do I check that I have enough samples (as in my case the sampling step is expensive)?

As illustration: In this next scenario I can visually see after 10 samples that the distributions are very likely to be different.

However, in the following scenario, more samples are required to make the same conclusion.

• How do you have just $X$ and $Y$ in your question yet three distinct CDFs in your plots?
– Dave
Dec 17, 2021 at 16:50
• @Dave the green CDF uses 10 random points from the 100 points that make up the orange CDF, so they are both constructed from $Y$
– Deb
Dec 17, 2021 at 17:01
• Okay, so you aim to compare just two distributions, right, not three?
– Dave
Dec 17, 2021 at 17:01

As with any power and sample size determination, the sample size to achieve given power depends on the 'distance' between the two models being compared.

The test statistic $$D$$ of the Kolmogorov-Smirnov test is the maximum vertical distance between the two ECDFs (plotted as step functions).

If you are using a pooled two-sample t test to compare means of two normal distributions (having equal variances), then there is an exact analytic formula, using a noncentral t distribution, to find the power for given $$n$$'s, significance level, common variance $$\sigma^2$$ and distance $$\Delta$$ between means to be detected.

I know of no such formula for the K-S test, but it is possible to use simulation to approximate the power for given sample size and significance level.

For example, suppose you want to detect the difference between samples of size $$n=10$$ from distributions $$\mathsf{Gamma}(\mathrm{shape}=4, \mathrm{rate}=2)$$ and $$\mathsf{Gamma}(6, 3).$$ Then CDFs are much the same and ten observations from each will probably not give good power at the 5% level.

A simulation using data from 10,000 samples of size 10 shows the power to be uselessly small.

set.seed(1217)
pv = replicate(10^4,ks.test(
rgamma(10,4,2),
rgamma(10,6,3))$p.val) mean(pv <= .05) [1] 0.0144 # aprx power  You would need sample sizes something like 1000 in order to have reasonable power (say 90%) to detect the difference between these two distributions. set.seed(1218) pv = replicate(10^4, ks.test( rgamma(1000,4,2), rgamma(1000,6,3))$p.val)
mean(pv <= .05)
[1] 0.9027 # aprx power


By contrast, if you are trying to distinguish between the distributions $$\mathsf{Exp}(\mathrm{rate}=2)$$ and $$\mathsf{Norm}(1,1),$$ then samples of size 20 might be good enough.

A test at the 5% level with samples of size twenty has almost 80% power.

set.seed(2021)
pv = replicate(10^4, ks.test(rexp(20,2),
rnorm(20,0, 1))\$p.val)
mean(pv <= .05)
[1] 0.7882


Unfortunately, the K-S test is not known for good power. If the differences between your distributions of interest are due only to different parameter values (e.g, distinguishing between two normal distributions or two exponential distributions), then you might find a parametric test using key parameters that gives better results.