1
$\begingroup$

From Wikipedia

A large amount of computing resources is required to compute exact probabilities for the Kruskal-Wallis test. Existing software only provides exact probabilities for sample sizes less than about 30 participants. These software programs rely on asymptotic approximation for larger sample sizes. Exact probability values for larger sample sizes are available. Spurrier (2003) published exact probability tables for samples as large as 45 participants. Meyer and Seaman (2006) produced exact probability distributions for samples as large as 105 participants. Critical value tables and exact probabilities from Meyer and Seaman are available for download at http://faculty.virginia.edu/kruskal-wallis/. A paper describing their work may also be found there.

I think exact probabilities are the accurate probabilities, but why does it say that "Existing software only provides exact probabilities for sample sizes less than about 30 participants. These software programs rely on asymptotic approximation for larger sample sizes"? Is it wrong?

Thanks and regards!

$\endgroup$
1
$\begingroup$

You're basically asking 'why did whoever wrote that section say that?' -- presumably they wrote what they did because they thought it was true.

It's clearly true of many software programs, and for the reason explicitly stated in the first sentence you quote - after all, unless you very carefully implement efficient algorithms for exact calculations of tail probabilities, the number of calculations grows extremely rapidly (well in fact it also grows very rapidly even if you do implement such calculations, it just lets you push the achievable $n$ up some distance).

Since you give the reason why the asymptotic distribution is used for large $n$ right in your question, I'm puzzled what else you're after here, except perhaps an explanation of why more software hasn't implemented the most efficient known algorithm.

$\endgroup$
  • $\begingroup$ Thanks! My confusion comes from that asymptotic approximation isn't accurate, but exact probability is accurate, and they can't coexist in one single run of a software program. $\endgroup$ – Tim May 29 '13 at 1:50
  • 2
    $\begingroup$ To add to your confusion: exact test are often not accurate at all: Agresti, Alan, and Brent A. Coull. (1998) "Approximate is better than "exact" for interval estimation of binomial proportions." The American Statistician 52(2): 119-126. $\endgroup$ – Maarten Buis May 29 '13 at 7:45
  • 1
    $\begingroup$ @MaartenBuis +1, Though in such issues come down to things like what is conditioned on, how you deal with the nuisance parameter in the binomial case, and what the sample size is; that particular argument has been going back and forth for about 80 years. If one takes the greater discreteness of the Fisher test into account when selecting a level (so if the level is chosen from achievable levels), the advantage to at least some binomial approaches disappears. $\endgroup$ – Glen_b -Reinstate Monica May 29 '13 at 7:51
  • 2
    $\begingroup$ @MaartenBuis By the way, Agresti seems to have put a copy of that paper up at his academic pages; it's an important paper on the issue and definitely worth a read (I first read it when it originally came out), but I think it needs to be considered as a part of a wider discussion. I'd certainly agree that exact tests are only exact in a particular sense and in particular circumstances. $\endgroup$ – Glen_b -Reinstate Monica May 29 '13 at 7:58
  • 1
    $\begingroup$ @Glen_b +1, sure. My comment was a reaction to the comment: "asymptotic approximation isn't accurate, but exact probability is accurate". I often see people being too impressed by any test carrying the label "exact". $\endgroup$ – Maarten Buis May 29 '13 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.