How can I reject or confirm the null hypothesis without p-value?

Question

I have two, large, two- or three-dimensional samples. I have used implementation of multidimensional Kolmogorov-Smirnov test in R (named Peacock.test) to compare these two samples and calculate test statistic. Is it any method to check null and alternative hypothesis without using p-value? How can I determine if the test statistic is in the critical area of the test?

Via Wikipedia, I can reject null hypothesis in two sample, one-dimensional KS test when: $$ D_{n,m} > c(\alpha) \sqrt{\frac{n+m}{nm}},$$ where $D_{n,m}$ is test statistic, $c(\alpha)$ I know from table, $n$ and $m$ are the sizes of samples. Is it works well for multidimensional Kolmogorov-Smirnov test?

Answer 1

The Kolmogorov-Smirnov test uses an approximating null distribution called the Kolmogorov distribution, which can be used to obtain an approximation to the p-value. This is the same distribution that is used to construct the tabular values of critical points for the test.

Let $X_1, ..., X_n$ and $Y_1, ..., Y_n$ be independent random variables with respective continuous distribution functions $F_X$ and $F_Y$. Let $E_X$ and $E_Y$ be the respective empirical distribution functions of the samples. The Kolmogorov-Smirnov test statistic is:

$$D_{n,m} \equiv \sup_{t} | E_X(t) - E_Y(t) |.$$

Let $K(t) \equiv 1 - \sum_{k=1}^\infty (-1)^{k-1} \exp (-2k^2 t^2)$ be the CDF of the Kolmogorov distribution. Then it can be shown that the Kolmogorov-Smirnov statistic obeys the following asymptotic result:

$$\lim_{n \rightarrow \infty} \lim_{m \rightarrow \infty} \mathbb{P} \Bigg( \sqrt{\frac{mn}{m+n}} \cdot D_{n,m} \leqslant t \Bigg| F_X = F_Y \Bigg) = K(t).$$

Hence, the p-value can be approximated by the large sample distribution as follows:

$$\begin{equation} \begin{aligned} p \equiv p (d_{n,m} ) &\equiv \mathbb{P}(D_{n,m} \geqslant d_{n,m} | F_X = F_Y) \\[6pt] &= 1 - \mathbb{P}(D_{n,m} < d_{n,m} | F_X = F_Y) \\[6pt] &\approx 1 - K \Bigg( \sqrt{\frac{mn}{m+n}} d_{n,m} \Bigg) \\[6pt] &= \sum_{k=1}^\infty (-1)^{k-1} \exp \Bigg( -2k^2 \frac{mn}{m+n} d_{n,m}^2 \Bigg). \end{aligned} \end{equation}$$

This expression can be evaluated numerically by truncation at a large value of $k$, yielding an approximate p-value for the two-sample Kolmogorov-Smirnov test. Statistical software will calculate this probability for you.