14
$\begingroup$

Given a reference distribution and an unknown sample, we need some statistical test to determine if the unknown sample came from the reference (one-sample test), or given two samples to determine whether they are realizations from the same distribution.

A very popular test for both is the one-sample Kolmogorov-Smirnov test, or two-sample K-S test.

However, it is well-known that the Kolmogorov-Smirnov test is most sensitive at the median location of the distribution, and much less sensitive when there are differences in the tails, which in many cases is the more interesting region (see this assertion in the link for Kuiper's test below).

There are other tests which are equally-sensitive at the tails as at the median, such as the Anderson-Darling test, or Kuiper's test, which also has the additional advantage of being applicable to cyclical shifts.

Other alternative test:

My question is: What are the advantages (if any) of the Kolmogorov-Smirnov test over other tests, such as Anderson-Darling, Kuiper's, and other alternatives? Please feel free to recommend any other test not mentioned in the question. Please clarify objective quantitative advantages.

(An answer could be, perhaps, "KS has no objective quantitative advantages besides popularity and mind-share", but then please clarify the objective quantitative advantages of any other test you recommend and why.)

$\endgroup$
12
  • 4
    $\begingroup$ +1 How are you measuring "popularity" of the K-S tests? What would a measure of acceptable "deprecation" of the K-S tests look like? $\endgroup$
    – Alexis
    Commented Mar 27, 2023 at 20:49
  • 3
    $\begingroup$ The Kullback-Leibler divergence isn't a test. Nor is the Wasserstein Distance. Those links operationalize those metrics and create a test. There are quite a few examples of researchers who propose a distributional test based on a. the normative difference between the empirical and theoretical CDFs and b. any weight function (if any) applied. $\endgroup$
    – AdamO
    Commented Mar 27, 2023 at 21:23
  • 3
    $\begingroup$ @Number You can propose any number of tests based on a. which normative distance you use between the F and the G, and b. the weight function if any for the "distance". Actually finding the operating characteristics of these tests takes some work, and there is no free lunch. The UMP test would require you to actually know the DFs to make parametric inference. $\endgroup$
    – AdamO
    Commented Mar 27, 2023 at 21:55
  • 1
    $\begingroup$ You might get your question reopened if you edit it to ask about what advantages the KS test has over specific alternative tests. $\endgroup$ Commented Mar 28, 2023 at 6:03
  • 3
    $\begingroup$ Thank you. I have voted to reopen, and upvoted. $\endgroup$ Commented Mar 28, 2023 at 8:04

2 Answers 2

3
+50
$\begingroup$

Bear in mind some general deficiencies of hypothesis tests of "nowhere dense" sets

To give some context to the many intelligent answers I expect this question will attract, I'm going to start by pointing out a big disadvantage of all tests of this general class, including the KS test, the AD test, and many other variants that test whether or not data comes from a specific distribution or parametric distributional family. These are all tests of a null hypothesis that is "nowhere dense", meaning that we are testing an extremely specific class of distributions in the null hypothesis compared to much larger class of distributions in the altnerative hypothesis. In most cases where we have data from a process, there may be some theoretical reason to think that a particular distributional form might hold approximately, but it is extremely rare to have reason to believe that a narrow distributional form would hold exactly. Even if we have good theoretical reason to think that a certain distributional form would hold (e.g., mean of IID data leading to a normal distribution via the CLT) the assumptions made in the theoretical derivation are usually only approximations to reality, and so the exact distribution at issue is usually still slightly different to the theoretical distribution. Typically this means that the null hypothesis is "always false" and so the test effectively becomes one where the p-value will converge to zero with enough data.

This is the primary problem that motivated the ASA statement on p-value and the general aversion that the statistics community has to classical hypothesis tests with a point-null hypothesis. Many statisticians are averse to this type of test because it is a test of a hypothesis that is so specific that it is a priori impossible, and so if the null hypothesis is not rejected, that is only because we lack enough data for adequate test power. For this reason, many statisticians prefer to use interval estimation methods like confidence intervals and credibility intervals that give an estimate for a range of values of the unknown quantity/parameter/distribution of interest, rather than testing against a specific case. The KS test, AD test, etc., are not easily emenable to conversion to a confidence interval over the function space of possible distribution functions. It is possible to get pointwise confidence bands using reasoning analogous to the KS test, but these usually only give confidence intervals for specific points, rather than confidence sets over the space of distribution functions.

You can of course draw distinctions between distributional tests of this general class, in terms of their power function against specific kinds of alternatives. As you've pointed out in your question, some of the tests are more powerful against certain kinds of alternatives than others, and it is possible to do a deep dive into this by comparing alternative tests with simulation analysis, etc. While this is possible ---and it is useful in understanding the relative merits of all these tests--- this does not get past the primary problem that occurs when we test a point-null hypothesis or a "nowhere dense" hypothesis in a large space. If we test a hypothesis that is so specific that it is "always false" then the test operates as it should, rejecting the null with enough data. The test therefore becomes primarily a test of how much data we have, which we already know.

It is worth noting that it is possible to amend any test of a point-null hypothesis or a "nowhere dense" hypothesis by imposing a non-zero "tolerance" for deviation from the stipulated class within the null hypothesis and amending the test statistic accordingly. With a bit of work the KS test can be amended in this way, as can the AD test and other distributional tests. This solves the problem of testing against a "nowhere dense" region (and is how I would recommend dealing with these types of tests) but it then means that there is some additional arbitrarity in how large you make your "tolerance" in the null hypothesis. Confidence intervals and other region-based estimators sidestep this deficiency in the first place by looking for a region-based estimator of the unknown object of interest rather than looking at the level of evidence of deviation of a stipulated set of values of that object.

$\endgroup$
5
  • 1
    $\begingroup$ The question was "What are the advantages (if any) of the Kolmogorov-Smirnov test over other tests, such as Anderson-Darling, Kuiper's, and other alternatives?" and not what are the problems of a point-null hypothesis. $\endgroup$
    – rep_ho
    Commented Apr 6, 2023 at 9:53
  • 1
    $\begingroup$ Hence the opening sentence, stating what this answer is for. $\endgroup$
    – Ben
    Commented Apr 6, 2023 at 11:42
  • 1
    $\begingroup$ @Ben Thanks. Your current answer convinces very effectively of the futility of using a single point-null hypothesis, i.e., a so-called simple hypothesis test. Furthermore, you propose using tests based on confidence intervals (frequentist) or credibility intervals (Bayesian) that give an estimate for a range of values of the unknown quantity/parameter/distribution of interest, i.e., a so-called composite hypothesis test. Would be highly appreciated: Can you please add to your answer specific references for such proposed tests based on confidence intervals or credibility intervals? Thanks $\endgroup$
    – Number
    Commented Apr 6, 2023 at 20:06
  • 1
    $\begingroup$ I've awarded the bounty, due to the interesting and important point-of-view, but have not accepted the answer, because I still hope for a more direct answer to the question and/or an extended answer from @Ben (i.e., add to his answer specific references for his proposed tests based on confidence intervals or credibility intervals) $\endgroup$
    – Number
    Commented Apr 10, 2023 at 21:13
  • $\begingroup$ @Number: No problem --- hopefully you'll get an answer that directly addresses your question. Since my answer is already tangential to the question, I do not propose to extend it at present (though you're welcome to ask a separate question for more information if you like). $\endgroup$
    – Ben
    Commented Apr 10, 2023 at 22:56
1
$\begingroup$

Note: I do not intend to accept my own answer, just wanted to provide info from an interesting relevant reference I just came across, which may interest people viewing this question.

The paper

N. Razali and Y.B. Wah (2011), "Power comparisons of Shapiro–Wilk, Kolmogorov–Smirnov, Lilliefors and Anderson–Darling tests", Journal of Statistical Modeling and Analytics, 2 (1): 21–33

Shows

"Power comparisons of these four tests were obtained via Monte Carlo simulation of sample data generated from alternative distributions that follow symmetric and asymmetric distributions. ... Results show that Shapiro-Wilk test is the most powerful normality test, followed by Anderson-Darling test, Lilliefors test and Kolmogorov-Smirnov test. However, the power of all four tests is still low for small sample size."

$\endgroup$
2
  • 1
    $\begingroup$ Simulation results depend on the setup of the simulation, and their generalizability to other setups is not obvious. Perhaps this is obvious, but still worth keeping in mind. $\endgroup$ Commented Jun 21, 2023 at 14:31
  • $\begingroup$ @RichardHardy Yes, true. Thanks for the important comment. Note that the paper did clarify their assumptions. As in anything mathematical, the best thing is an analytic proof. $\endgroup$
    – Number
    Commented Jun 26, 2023 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.