# Should we always use 100 samples for an equivalence test given the KS test size problems?

My problem:-

I'm writing a test to see if a series of numbers come from certain theoretical distributions. And I need a p value so that software can automatically accept or reject $$H_0$$ on an $$\alpha=0.05$$ basis.

The context:-

Does the 2-sample KS test work? If so, why is it so unintuitive?

Is the Kolmogorov-Smirnov-Test too strict if the sample size is large?

Kolmogorov–Smirnov test: p-value and ks-test statistic decrease as sample size increases

Is normality testing 'essentially useless'?

Is there a rule of thumb regarding effect size and the two sample KS test?

Given the number of unspecific KS sample size answers here, it seems that this is still a valid (open) problem.

My question:-

Given the context and learned comments, it appears that the KS test only holds for a small sample size, $$n$$. Yet I can't find any quantitative recommendation on this site for $$n$$. So if I have a total sample size of one million values, should I just randomly pick a hundred of them for the KS test?

• The number of answers testifies to how much confusion there is among site visitors about Normality testing. There is a strong consensus here on CV about how it works and when it's useful to apply it. See stats.stackexchange.com/questions/2492 (the fourth link in your list). Many of these tests are frequently misunderstood because the programs and tables created to implement them limit them to small samples. That's not a mathematical or computational issue. The proposal to randomly select a subsample has been explored extensively: you have a good answer here already.
– whuber
Apr 5 at 21:40

The KS test does what it is supposed to do, even when you have a million observations. When the null hypothesis is true, the KS test rejects the null about $$\alpha$$ of the time. What does happen is that, because of the large sample size, there is considerable power to reject slight deviations from the null hypothesis that might not seem to have practical significance.

Thus, use all of your points.

If you do not like that the test has such power because you are rooting for $$p>\alpha$$, you should think hard about if hypothesis testing is the right tool for your work. Hypothesis testing is extremely literal, and it is a feature, not a bug, that hypothesis testing can detect small deviations from the null hypothesis when the sample size is large. If you want to test if the data are "close" to the null hypothesis, then you might want to think about how to quantify "close" and consider methods like equivalence testing.

You may be interested in the links below.

Is normality testing 'essentially useless'?

Significance test for large sample sizes

EDIT

Let's see a simulation with $$100,000$$ observations per test (to save computing time).

library(ggplot2)
set.seed(2023)
N <- 1e5
R <- 5000
ps1 <- ps2 <- rep(NA, R)
for (i in 1:R){

# Simulate draws from N(0, 1)
#
x <- rnorm(N, 0, 1)

# KS test if the distribution is N(0, 1) or not, then save the p-value
#
ps1[i] <- ks.test(x, "pnorm", 0, 1)$p.value # Simulate draws from N(0.01, 1) # y <- rnorm(N, 0.01, 1) # KS test if the distribution is N(0, 1) or not, then save the p-value # ps2[i] <- ks.test(y, "pnorm", 0, 1)$p.value

if (i %% 75 == 0 | i < 5 | R - i < 5){
print(paste(
i/R*100,
"% complete",
sep = ""
))
}
}

d1 <- data.frame(
pvalue = c(ps1, ps2),
CDF = ecdf(ps1)(c(ps1, ps2)),
null = "True"
)
d2 <- data.frame(
pvalue = c(ps1, ps2),
CDF = ecdf(ps2)(c(ps1, ps2)),
null = "False"
)
d <- rbind(d1, d2)
ggplot(d, aes(x = pvalue, y = CDF, col= null)) +
geom_point() +
geom_abline(slope = 1, intercept = 0) When the nul hypothesis is true, despite there being a huge number of observations (you can bump it up to a million and get the same result), the distribution of p-values is $$U(0,1)$$ (the blue CDF), meaning that there is a probability of $$\alpha$$ of falsely rejecting a null hypothesis. For instance, when $$\alpha = 0.05$$, ecdf(ps1)(0.05) shows that the true null hypothesis is rejected $$4.8\%$$ of the time, just like is supposed to happen.

However, the red graph shows that the KS test indeed has solid power to reject a false null hypothesis, even one that is slightly false $$\big(N(0.01, 1)$$ is the true distribution, while the null hypothesis is that the distribution is $$N(0, 1)\big)$$, and ecdf(ps2)(0.05) shows the test to have a power of $$77.78\%$$ to reject the false null hypothesis.

Importantly, however, the ability to reject this false null hypothesis does not come at the expense of not rejecting a true null hypothesis. The KS test works exactly how a hypothesis test is meant to work.

• So what do you recommend for $n$? Apr 5 at 20:25
• @PaulUszak Use all of your observations. The test is doing exactly what it is supposed to be doing when it detects small deviations from the null hypothesis, and the test is not especially likely to reject a true null hypothesis just because the sample size is large. If you do not like that the test can detect these small deviations from the null hypothesis because you are rooting for $p>\alpha$, you should think hard about if hypothesis testing is the right tool for your work.
– Dave
Apr 5 at 20:25
• Downvoter ( @PaulUszak , maybe someone else), care to explain your objection?
– Dave
Apr 5 at 20:33
• My software wants to know what is $n$. Apr 6 at 4:44
• @PaulUszak You software should have some function that also counts the number of values in your variable, such as length.
– Dave
Apr 6 at 10:35