3
$\begingroup$

I want to compare the performance of $N$ methods on $m$ datasets. I performed the Friedman test and after that the Nemenyi test.

The Nemenyi test is though not powerful enough to conclude that there is a significant difference between the performance of the best performing method and the second performing method.

I then used the Wilcoxon method to compare the performance of these two models, and Wilcoxon method indeed returns that the difference in their performance is significant.

What can I conclude? Can I conclude that the difference in performance of the two models is statistically significant?

$\endgroup$

1 Answer 1

2
$\begingroup$

Per Wikipedia on Wilcoxon signed-rank test:

Assumptions

  • Data are paired and come from the same population.

  • Each pair is chosen randomly and independently[citation needed].

  • The data are measured on at least an interval scale when, as is usual, within-pair differences are calculated to perform the test (though it does suffice that within-pair comparisons are on an ordinal scale).

Now, the Friedman test, a commonly employed non-parametric test for complete block designs is related to the non-parametric Durbin test as it reduces to the Friedman test for complete block design scenarios. The cited underlying assumptions per a source are:

  • The b blocks are mutually independent. That means the results within one block do not affect the results within other blocks.

  • The data can be meaningfully ranked (i.e., the data have at least an ordinal scale).

The Nemenyi post-hoc test, however, has been discussed previously on this forum here, and the appropriate underlying assumptions are those that apply for the use of the studentized range distribution. Per Wikipedia, these are:

Assumptions

  • The observations being tested are independent within and among the groups.

  • The groups associated with each mean in the test are normally distributed.

  • There is equal within-group variance across the groups associated with each mean in the test (homogeneity of variance).

Note, the Nemenyi test has an implicit assumption normality of means. This assumption is absent elsewhere, as such, I would more likely suspect, that there is a non-normality issue. Also, per the references cited, be mindful of the 'family of test' significance level adjustment.

$\endgroup$
4
  • 1
    $\begingroup$ couldn't it be that the Nemenyi test is not powerful enough? I obtain CD (critical distance) equal to 1.64. Thus, even though the best performing algorithm is always ranked first and the second is always ranked second I get that there is no significant difference $\endgroup$
    – elena
    Commented May 26, 2020 at 20:57
  • $\begingroup$ Note, there is also a homogeneity of variance assumption in addition to normality of means. With these assumptions, I would hope for more power and not less. $\endgroup$
    – AJKOER
    Commented May 26, 2020 at 21:12
  • $\begingroup$ Here's another source (specifically for Friedman) stating that the blocks ("rows") must be independent: onlinelibrary.wiley.com/doi/epdf/10.1002/… $\endgroup$
    – Pro Q
    Commented Mar 6 at 21:10
  • $\begingroup$ To clarify, this should be interpreted as the answerer describes above, that each of the rows is independent from each of the other rows. $\endgroup$
    – Pro Q
    Commented Mar 6 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.