# A basic question about the Kolmogorov—Smirnov test and the $p$-value

I have a pretty basic doubt about the KS test (two samples). From what I understand:

We reject null hypothesis, under the confidence parameter $\alpha$, if the p-value (that is, the probability that the statistic takes a value $\geq$ observed, under the null hypothesis) is $\leq \alpha$ (Typically, $\alpha$ is set to 5%).

So, in particular, the bigger the $\alpha$, the more likely we'll reject.

Here, in the Kolmogorov—Smirnov our statistic is the Kolmogorov distance $X\stackrel{\rm def}{=}d_K(\hat{p},\hat{q})$, where $\hat{p},\hat{q}$ are the empirical distributions obtained from our two samples. We have a mapping $\alpha \mapsto c_\alpha$ (from the theoretical analysis of the distribution of $X$, under the assumption that our two unknown distributions $p,q$ are equal: $p=q$) The two-sample KS test will reject the null hypothesis ($H_0\colon p=q$) if $X\geq c_\alpha$.

From the above, it is clear that $c_\alpha$ decreases when $\alpha$ increases.

Now, my question:

Can I argue in any way about the guarantees I get for rejecting the alternative hypothesis, $p\neq q$? In other terms, what can I say from that $X$ about using $p\neq q$ as the null hypothesis $H_0$, instead of $p=q$?

• Null hypothesis must be simple hypothesis, like $p = q$, because otherwise the distribution of test statistics cannot be determined. $p \ne q$ is a composite hypothesis and cannot be used as null. – user158565 Apr 29 '17 at 21:25
• @a_statistician It's quite possible to have composite nulls. The really big difficulty here is with the point alternative. – Glen_b Apr 30 '17 at 7:47
• @Clement we have a number of posts on site about trying to put $\neq$ into the null of hypothesis tests. See whuber's comments here for example (sorry I can't find a question with that information in an actual answer. – Glen_b Apr 30 '17 at 7:50
• @Glen_b OK, in this case, what is the distribution of $(X|p \ne q)$? – user158565 Apr 30 '17 at 15:19
• @a_statistician That would be a case with a point alternative, which is where I said there'd be a problem. For a composite null to work, you need a composite alternative at least (and preferably the equality under the null). You don't have to have $\alpha$ associated with every point in the null, you need to be no higher than $\alpha$ everywhere. – Glen_b Apr 30 '17 at 15:41