1
$\begingroup$

Let's say I have samples $X$ from one distribution and I am now collecting new samples $Y$ from another unknown distribution. What would be the best test to check whether the samples $Y$ come from the same distribution of $X$? Should I use the Kolmogorov-Smirnov test? And how do I check that I have enough samples (as in my case the sampling step is expensive)?

As illustration: In this next scenario I can visually see after 10 samples that the distributions are very likely to be different.

enter image description here

However, in the following scenario, more samples are required to make the same conclusion.

enter image description here

$\endgroup$
3
  • $\begingroup$ How do you have just $X$ and $Y$ in your question yet three distinct CDFs in your plots? $\endgroup$
    – Dave
    Dec 17, 2021 at 16:50
  • $\begingroup$ @Dave the green CDF uses 10 random points from the 100 points that make up the orange CDF, so they are both constructed from $Y$ $\endgroup$
    – Deb
    Dec 17, 2021 at 17:01
  • $\begingroup$ Okay, so you aim to compare just two distributions, right, not three? $\endgroup$
    – Dave
    Dec 17, 2021 at 17:01

1 Answer 1

2
$\begingroup$

As with any power and sample size determination, the sample size to achieve given power depends on the 'distance' between the two models being compared.

The test statistic $D$ of the Kolmogorov-Smirnov test is the maximum vertical distance between the two ECDFs (plotted as step functions).

If you are using a pooled two-sample t test to compare means of two normal distributions (having equal variances), then there is an exact analytic formula, using a noncentral t distribution, to find the power for given $n$'s, significance level, common variance $\sigma^2$ and distance $\Delta$ between means to be detected.

I know of no such formula for the K-S test, but it is possible to use simulation to approximate the power for given sample size and significance level.

For example, suppose you want to detect the difference between samples of size $n=10$ from distributions $\mathsf{Gamma}(\mathrm{shape}=4, \mathrm{rate}=2)$ and $\mathsf{Gamma}(6, 3).$ Then CDFs are much the same and ten observations from each will probably not give good power at the 5% level.

enter image description here

A simulation using data from 10,000 samples of size 10 shows the power to be uselessly small.

set.seed(1217)
pv = replicate(10^4,ks.test(
        rgamma(10,4,2), 
        rgamma(10,6,3))$p.val)
mean(pv <= .05)
[1] 0.0144  # aprx power

You would need sample sizes something like 1000 in order to have reasonable power (say 90%) to detect the difference between these two distributions.

set.seed(1218)
pv = replicate(10^4, ks.test(
      rgamma(1000,4,2), 
      rgamma(1000,6,3))$p.val)
mean(pv <= .05)
[1] 0.9027 # aprx power

By contrast, if you are trying to distinguish between the distributions $\mathsf{Exp}(\mathrm{rate}=2)$ and $\mathsf{Norm}(1,1),$ then samples of size 20 might be good enough.

enter image description here

A test at the 5% level with samples of size twenty has almost 80% power.

set.seed(2021)
pv = replicate(10^4, ks.test(rexp(20,2), 
                 rnorm(20,0, 1))$p.val)
mean(pv <= .05)
[1] 0.7882

Unfortunately, the K-S test is not known for good power. If the differences between your distributions of interest are due only to different parameter values (e.g, distinguishing between two normal distributions or two exponential distributions), then you might find a parametric test using key parameters that gives better results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.