I want to compare two samples with a Kolmogorov-Smirnov test. Wikipedia states the null hypothesis is rejected at $\alpha$ if:
$\sqrt( \frac{nn'}{n+n'}) D_{nn'} > K_\alpha$
where $n$ and $n'$ are the sizes of samples, D the KS-statistic and $K_\alpha$ the critical value (probably everyone here already knows). I wonder about the sample sizes: according to this formula every null hypothesis is rejected, if the samples are just large enough.
Could anybody enlighten me, what I am misunderstanding?
It is correct that every result will result in rejecting the null if the sample is large enough. This is true, not just for test using KS, but for any significance test. On one level, this is very reasonable: As sample size increases, our estimates get more precise. At some point, they will get precise enough that any difference $d$ will be large enough to be different from $0$ at a level $\alpha$ .
However, it is also one of the fundamental problems with significance testing: The p-value does not (in the vast majority of cases) tell you what you want to know. That is, the question you are answering is this one:
If, in the population from which this sample was drawn there really was no effect at all, how likely is it that, in a sample of the size we have, the test statistic would be as large as the one we got, or larger?
The simple alternative is to pay more attention to effect sizes and confidence intervals. The complex alternative is to become a Bayesian. :-).
