If the model does not satisfy ANOVA assumptions (normality in particular), if one-way, Kruskal-Wallis non-parametric test is recommended. But, what if you have multiple factors?


You can use a permutation test.

Form your hypothesis as a full and reduced model test and using the original data compute the F-statistic for the full and reduced model test (or another stat of interest).

Now compute the fitted values and residuals for the reduced model, then randomly permute the residuals and add them back to the fitted values, now do the full and reduced test on the permuted dataset and save the F-statistic (or other). Repeate this many times (like 1999).

The p-value is then the proportion of the statistics that are greater than or equal to the original statistic.

This can be used to test interactions or groups of terms including interactions.

  • 6
    $\begingroup$ For a discussion of different permutation strategies in factorial ANOVA-designs, see e.g. avesbiodiv.mncn.csic.es/estadistica/permut1.pdf (pdf) $\endgroup$
    – caracal
    Jun 21 '11 at 11:40
  • 3
    $\begingroup$ This works, but what happens to the power of the test? E.g., even if there is only one (far) outlying value and the rest of the residuals are normally distributed, it appears that using the F-statistic may have little power in the permutation test to detect anything. The paper referenced by @caracal discusses the subtleties and assesses when the F-statistic approach works and when it might fail. $\endgroup$
    – whuber
    Jun 21 '11 at 14:31
  • $\begingroup$ "The p-value is then the proportion of the statistics that are greater than or equal to the original statistic" --> to the original statistic calculated on the full model. correct? $\endgroup$ Nov 8 '11 at 2:57
  • 1
    $\begingroup$ @toto_tico, using ranks is one option for non-parametric tests, but is not the only one (permutation testing is another that does not rely on ranks). Combining factors into a single factor works if you want to test all-or-nothing, but does not work for testing if the interaction is significant beyond the effects of the main effects, or testing one factor given the other factor is in the model. $\endgroup$
    – Greg Snow
    Nov 13 '15 at 16:57
  • 1
    $\begingroup$ @toto_tico, just code it directly. See the example I added based on your other comment (stats.stackexchange.com/questions/41199/…). $\endgroup$
    – Greg Snow
    Nov 16 '15 at 17:33

The Kruskal-Wallis test is a special case of the proportional odds model. You can use the proportional odds model to model multiple factors, adjust for covariates, etc.

  • 7
    $\begingroup$ If one would like to learn more about the connection between K-W and the proportional odds model, what would be a good reference? $\endgroup$
    – whuber
    Jun 21 '11 at 14:24
  • 8
    $\begingroup$ @ARTICLE{pet89ord, author = {Peterson, Bercedis}, year = 1989, title = {Re: {Ordinal} regression models for epidemiologic data}, journal = Am J Epi, volume = 129, pages = {745-748}, annote = {proportional odds model; partial proportional odds} } @ARTICLE{mcc80reg, author = {{McCullagh}, Peter}, year = 1980, title = {Regression models for ordinal data}, journal = JRSSB, volume = 42, pages = {109-142}, annote = {ordinal logistic model} }See also Whitehead Stat in Med 1993 p. 2257 $\endgroup$ Jun 22 '11 at 2:36
  • 3
    $\begingroup$ I've never seen someone post bibtex before on a stackexchange site. $\endgroup$
    – abalter
    Mar 24 '20 at 1:41

Friedman's test provides a non-parametric equivalent to a one-way ANOVA with a blocking factor, but can't do anything more complex than this.

  • $\begingroup$ Friedman's test is low power. $\endgroup$ Mar 26 '20 at 23:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.