Has two one-sided tests for equivalence (TOST) been framed for the Kolmogorov–Smirnov test to test the negativist null hypothesis that two distributions differ by at least some researcher-specified level?

If not TOST, then some other form of equivalence test?

Nick Stauner wisely points out that (I should already know ;) that there are other nonparametric TOST equivalence tests for null hypotheses for stochastic equivalence, and, with more restrictive assumptions, for median equivalence.


2 Answers 2


Ok, here's my first attempt. Close scrutiny and comments appreciated!

The Two-Sample Hypotheses
If we can frame two-sample one-sided Kolmogorov-Smirnov hypothesis tests, with null and alternate hypotheses along these lines:

$\text{H}_{0}\text{: }F_{Y}\left(t\right) \geq F_{X}\left(t\right)$, and

$\text{H}_{\text{A}}\text{: }F_{Y}\left(t\right) < F_{X}\left(t\right)$, for at least one $t$, where:

  • the test statistic $D^{-}=\left|\min_{t}\left(F_{Y}\left(t\right) - F_{X}\left(t\right)\right)\right|$ corresponds to $\text{H}_0\text{: }F_{Y}\left(t\right) \geq F_{X}\left(t\right)$;

  • the test statistic $D^{+}=\left|\max_{t}\left(F_{Y}\left(t\right) - F_{X}\left(t\right)\right)\right|$ corresponds to $\text{H}_0\text{: }F_{Y}\left(t\right) \leq F_{X}\left(t\right)$; and

  • $F_{Y}\left(t\right)$ & $F_{X}\left(t\right)$ are the empirical CDFs of samples $Y$ and $X$,

then it should be reasonable to create a general interval hypothesis for an equivalence test along these lines (assuming that the equivalence interval is symmetric for the moment):

$\text{H}^{-}_0\text{: }\left|F_{Y}\left(t\right) - F_{X}\left(t\right)\right| \geq \Delta$, and

$\text{H}^{-}_{\text{A}}\text{: }\left|F_{Y}\left(t\right) - F_{X}\left(t\right)\right| < \Delta$, for at least one $t$.

This would translate to the specific two one-sided "negativist" null hypotheses to test for equivalence (these two hypotheses take the same form, since both $D^{+}$ and $D^{-}$ are strictly non-negative):

$\text{H}^{-}_{01}\text{: }D^{+} \geq \Delta$, or

$\text{H}^{-}_{02}\text{: }D^{-} \geq \Delta$.

Rejecting both $\text{H}^{-}_{01}$ and $\text{H}^{-}_{02}$ would lead one to conclude that $-\Delta < F_{Y}\left(t\right) - F_{X}\left(t\right) < \Delta$. Of course, the equivalence interval need not be symmetric, and $-\Delta$ and $\Delta$ could be replaced with $\Delta_{2}$ (lower) and $\Delta_{1}$ (upper) for the respective one-sided null hypotheses.

The Test Statistics (Updated: Delta is outside the absolute value sign)
The test statistics $D^{+}_{1}$ and $D^{-}_{2}$ (leaving the $n_{Y}$ and $n_{X}$ implicit) correspond to $\text{H}^{-}_{01}$ and $\text{H}^{-}_{02}$, respectively, and are:

$D^{+}_{1} = \Delta - D^{+} = \Delta - \left|\max_{t}\left[\left(F_{Y}\left(t\right) - F_{X}\left(t\right)\right)\right]\right|$, and

$D^{-}_{2} = \Delta - D^{-} = \Delta - \left|\min_{t}\left[\left(F_{Y}\left(t\right) - F_{X}\left(t\right)\right)\right]\right|$

The Equivalence/Relevance Threshold
The interval $[-\Delta, \Delta]$—or $[\Delta_{2}, \Delta_{1}]$, if using an asymmetric equivalence interval—is expressed in units of $D^{+}$ and $D^{-}$, or the magnitude of differenced probabilities. As $n_{Y}$ and $n_{X}$ approach infinity, the CDF of $D^{+}$ or $D^{-}$ for $n_{Y},n_{X}$ approaches $0$ for $t\le 0$, and must be $>0$ for $t > 0$:

$$\lim_{n_{Y},n_{X}\to \infty}p^{+} = \text{P}\left(\sqrt{\frac{n_{Y}n_{X}}{n_{Y}+n_{X}}}D^{+} \le t\right) = 1 - e^{-2t^{2}}$$

CDF of D^+ (or D^-)

So it seems to me that the PDF for sample size-scaled $D^{+}$ (or sample size-scaled $D^{-}$) must be $0$ for $t<0$, and must be $>0$ for $t \ge 0$:

$$f(t) = {1 - e^{-2t^{2}}}\frac{d}{dt} = 4te^{-2t^{2}}$$

PDF of D^+ (or D^-)

Glen_b points out that this is a Rayleigh distribution with $\sigma=\frac{1}{2}$. So the large sample quantile function for sample size-scaled $D^{+}$ and $D^{-}$ is:

$$\text{CDF}^{-1} = Q\left(p\right) = \sqrt{\frac{-\ln{\left(1 - p\right)}}{2}}$$

and a liberal choice of $\Delta$ might be the critical value $Q_{\alpha}+\sigma/2 = Q_{\alpha}+\frac{1}{4}$, and a more strict choice the critical value $Q_{\alpha}+\sigma/4=Q_{\alpha}+\frac{1}{8}$.

  • 1
    $\begingroup$ In the line where you pass from the cdf to the pdf, I think you got that wrong. Let $K_{n_{Y},n_{X}}=\sqrt{\frac{n_{Y}n_{X}}{n_{Y}+n_{X}}}D^{+}$, so (abusing notation), in the limit $P(K_{\infty,\infty}\leq t) = 1 - e^{-2t^{2}}$. Then $f_K(t) = \frac{d}{dt} 1 - e^{-2t^2} = 4t\, e^{-2t^2}\quad$ (note the $t$ after the $4$). (note also a missing sign in the exponent in the line above the taking of the derivative. Also I'm not sure why you have an integral symbol there, but maybe I misunderstood something.) $\endgroup$
    – Glen_b
    Jul 19, 2014 at 10:13
  • 2
    $\begingroup$ @stochazesthai $D_{1}$ and $D_{2}$ are two one-sided test statistics. Per TOST you need to reject both the null hypotheses to which these test statistics apply. $Q_{\alpha}$ is a critical value from CDF$^{-1}$ on the above line, and where you want to sub in $1-\alpha$ for $p$ (e.g. $Q_{\alpha} = \sqrt{\frac{-\ln{(1-(1-\alpha))}}{2}}$). The choice of $\Delta$ depends on how far past $Q_{\alpha}$ (the critical rejection value for a plain old positivist $H_{0}$) you need to go, before you conclude relevant difference (e.g. liberal 'equivalence' is $\frac{1}{4}$ $\sigma$ beyond $Q_{\alpha}$). $\endgroup$
    – Alexis
    May 18, 2015 at 20:02
  • 2
    $\begingroup$ @stochazesthai (Continuing) So if both $D_{1} \ge \Delta$ and $D_{2} \ge \Delta$, then you reject $H_{0}^{-}$. $\endgroup$
    – Alexis
    May 18, 2015 at 20:07
  • 2
    $\begingroup$ @stochazesthai Whoops! I should have put the quotes around the word liberal rather than equivalence two comments back. :) $\endgroup$
    – Alexis
    May 18, 2015 at 22:42
  • 2
    $\begingroup$ @stochazesthai If $D_{1} \ge \Delta$, then reject $H^{-}_{01}$, if $D_{1} < \Delta$, then fail to reject $H^{-}_{01}$. If $D_{2} \ge \Delta$, then reject $H^{-}_{02}$, if $D_{2} < \Delta$, then fail to reject $H^{-}_{02}$. If reject both $H^{-}_{01}$ and $H^{-}_{02}$, then reject $H^{-}_{0}$, otherwise fail to reject $H^{-}_{0}$. $\endgroup$
    – Alexis
    May 19, 2015 at 6:45

An alternative to TOST in equivalence testing is based on the confidence interval approach:

Let $\Delta$ denote the prespecified equivalence margin and
$$ \theta := \sup_t |F_X(t) - F_Y(t)| $$ the Kolmogorov-Smirnov distance between the unknown underlying distribution functions.

Now, if a 90% confidence interval for $\theta$ is completely within $[-\Delta, \Delta]$, then we may be 95% certain that $\theta$ is enough close to 0 to speak of "equivalence".

Without knowing the underlying distributions, it seems to be hopeless to derive an approximate analytic confidence interval, so we might need to rely on (bias corrected) bootstrap confidence intervals based on resampling from pairs $X$ and $Y$. (I don't want to find conditions for their validity in this particular application though...)

  • $\begingroup$ Excellent. Do you have a citation for anyone undertaking the CI of $D_{n_{1},n_{2}}$ (bootstrap or otherwise)? $\endgroup$
    – Alexis
    Jul 20, 2014 at 21:03
  • 1
    $\begingroup$ Good point... The short paper tomswebpage.net/images/K-S_test.doc mentions the "Handbook of Parametric and Nonparametric Statistical Procedures, Fifth Edition by David J.Sheskin (Apr 27, 2011)." to offer a two-sample case construcion for D. But at the moment, I don't have access to this book. $\endgroup$
    – Michael M
    Jul 21, 2014 at 5:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.