3
$\begingroup$

In Statistical Comparisons of Classifiers over Multiple Data Sets (p. 12) Janez Demsar suggests to use the Bonferroni-Dunn test as a post-hoc test after the null hypothesis of an initial Friedman test was rejected.

Questions:

  1. Friedman test is non-parametric. May one use a parametric test as a post-hoc test (e.g., paired t-test with Bonferroni adjustment)?
  2. Is the Wilcoxon signed-rank test with Bonferroni adjustment also an appropriate post-hoc test in this case?
  3. The paper does not suggest pairwise comparisons of all groups if one wants to test a new algorithm against existing baselines. Instead, one should only compare the new algorithm (control) to the existing ones. How would Bonferroni adjustment work in this case? Still by dividing by the number of tests that were conducted?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

None of the tests mentioned (t-test, Wilcoxon signed rank test, Dunn test) match the Friedman's test, with how they handle the data or the assumptions involved. There are post-hoc test specifically for Friedman's test. The best source I can suggest is the documentation for the R package PMCMRplus cran.r-project.org/web/packages/PMCMRplus/PMCMRplus.pdf, and look at the functions that start with frd. These tests have names like Conover, Miller, Nemenyi, Siegel and Castellan, but be cautious because there may be distinct tests attributed to these authors.

You also might reconsider using Friedman's test. There may be better options for your purpose, like Quade test, ordinal regression, or aligned ranks transformation anova.

Improve this answer
$\endgroup$
6
  • $\begingroup$ taking your answer into consideration: if the omnibus test is non-parametric, using a parametric post-hoc test is bad idea, right? $\endgroup$
    – user378615
    Commented Jan 28, 2023 at 18:22
  • $\begingroup$ I think so. But maybe more importantly, the post-hoc test should "match" the omnibus test in terms of the assumptions and how data are handled. Using "mismatched" tests may not be wrong per se, but it would make me wonder why the author chose that omnibus test in the first place. $\endgroup$
    – Sal Mangiafico
    Commented Jan 28, 2023 at 18:29
  • $\begingroup$ alright. two further questions: do you no by heart 1. the omnibus test for Wilcoxon signed rank test 2. why Friedman and Wilcoxon signed rank test and do not match in their assumption (which assumption is different/violated)? $\endgroup$
    – user378615
    Commented Jan 28, 2023 at 18:43
  • $\begingroup$ Quade test is more similar to the signed rank test, in that the data are treated as interval. And Friedman test is more similar to the sign test, in that the data are treated as ordinal. $\endgroup$
    – Sal Mangiafico
    Commented Jan 28, 2023 at 18:57
  • $\begingroup$ Ok, another question because your answer are very precise and helpful: when I want to use a t-test (or any other test that needs familywise error correction) to compare only a single control to multiple other groups (basically question 3 from my post), do i still divide by the number of tests conducted? $\endgroup$
    – user378615
    Commented Jan 28, 2023 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.