12
$\begingroup$

I'm reading web pages for goodness of fit tests, when I came to the Anderson–Darling test and the Cramér–von Mises criterion.

So far I got the point; it seems the Anderson–Darling test and the Cramér–von Mises criterion are similar, just based on a different weighting function $w$. Also there's a variant of the Cramér–von Mises criterion named the Watson test.

Basically I have two questions here

  1. There are not many Google results about these two methods; are they still state-of-the-art? or replaced by some better approaches already?

    It's a bit of a surprise, as according to this paper on power comparisons of Shapiro–Wilk, Kolmogorov–Smirnov, Lilliefors and Anderson-Darling tests, AD is performing quite well; always better than Lilliefors and KS, and very close to the SW test, which is specifically designed for the normal distribution.

  2. What is the confidence interval for such tests?

    For the AD, CM and Watson tests, I saw the test statistics variable defined on the wiki pages, but didn't find the confidence interval.

    Things are just more straightforward for the KS test: on the wiki page, the confidence interval is defined by $K_\alpha$, which is defined from the cumulative distribution function of $K$.

$\endgroup$
0

2 Answers 2

8
$\begingroup$

There can be no single state-of-the-art for goodness of fit (for example no UMP test across general alternatives will exist, and really nothing even comes close -- even highly regarded omnibus tests have terrible power in some situations).

In general when selecting a test statistic you choose the kinds of deviation that it's most important to detect and use a test statistic that is good at that job. Some tests do very well at a wide variety of interesting alternatives, making them decent default choices, but that doesn't make them "state of the art".

The Anderson Darling is still very popular, and with good reason. The Cramer-von Mises test is much less used these days (to my surprise because it's usually better than the Kolmogorov-Smirnov, but simpler than the Anderson-Darling -- and often has better power than it on differences "in the middle" of the distribution)

All of these tests suffer from bias against some kinds of alternatives, and it's easy to find cases where the Anderson-Darling does much worse (terribly, really) than the other tests. (As I suggest, it's more 'horses for courses' than one test to rule them all). There's often little consideration given to this issue (what's best at picking up the deviations that matter the most to me?), unfortunately.

You may find some value in some of these posts:

Is Shapiro–Wilk the best normality test? Why might it be better than other tests like Anderson-Darling?

2 Sample Kolmogorov-Smirnov vs. Anderson-Darling vs Cramer-von-Mises (about two-sample tests but many of the statements carry over

Motivation for Kolmogorov distance between distributions (more theoretical discussion but there are several important points about practical implications)


I don't think you'll be able to form a confidence interval for the cdf in the Cramer-von Mises and Anderson Darline statistics, because the criteria are based on all of the deviations rather than just the largest.

$\endgroup$
7
  • $\begingroup$ I took "state of the art" to mean something that finds use that is not obsolescent. The existence of multiple goodness-of-fit definitions should signal to us that goodness-of-fit is not a single concept. Consider that "good" depends on "why" we are performing regression. Suppose we are fitting Model A to data B to obtain a best predictor of effect C. Then "good" is the best predictor of C not B. However, most often the question of how B and C differ is ignored. $\endgroup$
    – Carl
    Commented Aug 22, 2016 at 17:12
  • 1
    $\begingroup$ @Carl you may want to check a dictionary (or wikipedia) on what state of the art is usually taken to mean -- your interpretation of the phrase is not how most people read the phrase. Dictionaries say things like this: "the most recent stage in development, incorporating the newest ideas" and "the highest level of development at a given time" and "cutting edge, using the latest technology". In this context -- testing goodness of fit -- the phrase implies "the best we can possibly do right now". I insist that's not something you can really say about any single test. ... ctd $\endgroup$
    – Glen_b
    Commented Aug 22, 2016 at 21:44
  • 2
    $\begingroup$ ... e.g. We can say that popular tests like the Shapiro-Wilk (while very popular in testing normality) have competitors with broadly better power (e.g. see Shapiro&Chen 1995)-- but not in every situation. There's no single best choice of test (and hence, no actual 'state of the art'). Certainly I agree that what is best (state of the art) depends on circumstances --- that's the point of my answer; the possible answers are myriad - something good in one situation may be very poor in another. It pays to know when tests perform well rather than ask for "what's best" as if it were a single thing. $\endgroup$
    – Glen_b
    Commented Aug 22, 2016 at 21:47
  • $\begingroup$ True, your definition is more correct. However, there are many more methods than tests of methods, and the "state of the art" is largely fiction, i.e., the "art" has no "state" all it has are protagonists. Any response to such a nebulous posit is equivocal. I said 'yes' and you said 'no' and we both said the same thing. $\endgroup$
    – Carl
    Commented Aug 23, 2016 at 1:47
  • $\begingroup$ BTW, the question was "state of the art" or "replaced" which I took to mean "obsolete, or not obsolete". So there was a context for my answer which context was "Please assume that 'state of the art' and 'replace' are antonyms, and please choose one of those." You are correct that those are not antonyms, I was answering in context and you chose to beg the question. So, mine was the polite answer. And, I am going to vote for your answer, because I think it informative, if not overly polite. $\endgroup$
    – Carl
    Commented Aug 24, 2016 at 2:04
3
$\begingroup$

The Anderson-Darling test is not available on all distributions but has power that is good and close to the power for the Shapiro-Wilk test except for small numbers of samples so that the two are equivalent at $n=400$ Razali NM, Wah YB. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. Journal of Statistical Modeling and Analytics. 2011;2:21-33. However, the Shapiro-Wilk test is only for normal distribution testing. The Cramér–von Mises test and Pearson Chi-squared are general for all distribution fits to histograms and I think that Cramér–von Mises test has more power than Pearson Chi-squared. The Cramér–von Mises test is often more powerful cumulative density function goodness-of-fit test than the Kolmogorov-Smirnov test for a broad class of alternative hypotheses see 1. and can have power greater or less than t-testing. Chi-squared has difficulty with low cell counts, so range restrictions are used for fitting tails.

**Question 1: ... are ... these two methods... still state-of-the-art? or replaced by some better approaches already? Question 2 What is the confidence interval for such tests? **

Answer: They are state of the art. However, sometimes we want confidence intervals not probabilities. When comparing these methods to each other we speak of power rather than confidence intervals. Sometimes goodness-of-fit is analyzed using AIC, BIC and other criteria as contrasted to probabilities of good fitting, and sometimes the goodness-of-fit criterion is irrelevant, for example, when goodness-of-fit is not the criterion for fitting. In the latter case, our regression target may be a physical quantity not related to fitting, e.g., see Tk-GV.

(1) See, e.g., Section 5 in Stephens (1974) or page 110 in D’Agostino and Stephens (1986). Observe this is only empirical evidence for certain scenarios. Stephens, M. A. 1974. “EDF Statistics for Goodness of Fit and Some Comparisons.” Journal of the American Statistical Association 69 (347): 730–37. https://doi.org/10.2307/2286009. D’Agostino, R. B., and M. A. Stephens, eds. 1986. Goodness-of-Fit Techniques. Vol. 68. Statistics: Textbooks and Monographs. New York: Marcel Dekker. https://www.routledge.com/Goodness-of-Fit-Techniques/DAgostino/p/book/9780367580346.

$\endgroup$
11
  • 2
    $\begingroup$ NB The Anderson-Darling test is a weighted version of the Cramer-von Mises test; &, like it, suitable for any continuous distribution. $\endgroup$ Commented Aug 28, 2016 at 18:06
  • 1
    $\begingroup$ he Cramér–von Mises test is a more powerful cumulative density function goodness-of-fit test than the Kolmogorov-Smirnov test is incorrect. One is more powerful against some alternatives, the other against other alternatives. $\endgroup$ Commented Jun 22, 2023 at 17:59
  • $\begingroup$ @RichardHardy Thanks for your input, Richard. Can you provide some guidance as to when which is better? $\endgroup$
    – Carl
    Commented Jun 22, 2023 at 19:04
  • 1
    $\begingroup$ @RichardHardy Agreed. However, the information concerning tests that only apply to specific distributions and those that are more general is not in question. Certain general rules also apply, for example, a nonparametric test may apply to more distribution types at the cost of some sensitivity for each of them. So, it is not as if looking at the algorithms themselves was a waste of time, in fact, the devil in the details includes not only the data, but the algorithms themselves. $\endgroup$
    – Carl
    Commented Jun 23, 2023 at 15:04
  • 1
    $\begingroup$ I think this has improved your answer. Thank you for taking the time! $\endgroup$ Commented Jun 23, 2023 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.