1
$\begingroup$

This post does not include an answer for my question.

I have perform a Friedman test on a results obtained by several methods on multiple datasets, as described in this paper: "Statistical Comparisons of Classifiers over Multiple Data Sets"

The Friedman's statistic is (16.667) which exceeds the Chi-Square critical value (14.067) so the null hypothesis H0 (no difference between the methods) must be rejected. However, when I used a post-hoc test (Shaffer test) to identify the differences, All the adjusted P-values obtained by Shaffer test are bigger than level of significance 0.05. This means that there are no statically significant differences between the results obtained by these methods.

Is this result correct or not? Why the H0 is rejected at the first stage, then not rejected at the post-hoc test?

If this result is correct, what does it mean?

Original Data:

    Data sets,Method1, Method2, Method3, Method4, Method5, Method6, Method7, Method8
    F1,1.3708E-01,1.0227E-01,1.2801E-01,1.0512E-01,1.4476E-01,1.1309E-01,1.4002E-01,1.2801E-01
    F2,2.2731E-01,2.2758E-01,2.1174E-01,2.2516E-01,2.1865E-01,2.1740E-01,2.2620E-01,2.1174E-01
    F3,8.6773E-02,8.5749E-02,8.1918E-02,8.5238E-02,8.7778E-02,8.3436E-02,8.9080E-02,8.1918E-02
    F4,1.7313E-01,1.7626E-01,1.6378E-01,1.8294E-01,2.0685E-01,8.0293E-02,2.0000E-01,1.6378E-01

Average Ranks:

Method1  3.25
Method2  4.25
Method3  6.5
Method4  4.75
Method5  2.25
Method6  6.5
Method7  2
Method8  6.5

Adjusted P-values:

0.262494
0.262494
0.262494
0.296897
0.296897
0.296897
1.272641
1.272641
1.272641
1.797619
2.382635
3.102894
3.102894
3.102894
3.102894
3.22677
3.747857
3.747857
3.747857
3.747857
3.763891
3.94592
3.94592
3.94592
3.94592
3.94592
3.94592
3.94592

Addition information

According to that paper, "If the null-hypothesis is rejected, we can proceed with a post-hoc test." A a post-hoc test is used to identify which pairs of algorithms are significantly different than each other. Thus, in a multiple comparison, all possible pairwise comparisons need to be computed (N × N comparison). The number of possible comparisons in an all pairwise comparisons is N(N-1)/2. So we have 8(8 - 1)/2=28 hypotheses and an adjusted P-Value for each hypothes.

$\endgroup$
1
  • 1
    $\begingroup$ For all that you deny the relevance of the linked question, a number of points made there are relevant here. $\endgroup$
    – Glen_b
    Oct 1, 2021 at 5:13

1 Answer 1

1
$\begingroup$

Friedman's test and the ad hoc test use somewhat different distributions and criteria. In any case, Bonferroni ad hoc tests can be too conservative (not likely enough to reject), especially if you are making several ad hoc comparisons.

You do not give the P-values, the number of levels, level means/medians for the Friedman test, or ad hoc adjusted P-values. So it is not possible to give more than a guess at he particulars, trying to answer your question.

Here is my guess: It looks as if the main Friedman test may have been significant at something like the 2% significance level with maybe 8 levels of the main factor, and the most nearly significant ad hoc test may have been significant at something like the 7% level (adjusted). If so, the results may be annoying, but not wrong--or even entirely unexpected.

qchisq(.95, 7)
[1] 14.06714
1 - pchisq(16.67, 7)
[1] 0.01965259

If the overall Friedman test is significant, then it is reasonable to guess that the two levels with the greatest difference in location must be significantly different, but you may not be able to test for sure how locations of the (8 or so) levels are ranked.

In doing a 'power and sample size' analysis, in advance of taking data, one should try to determine how many 'blocks' are needed for ad hoc tests with a desired effect size to be detectable. This is not always easy to do.

Addendum. I see that you have appended a couple of dozen "adjusted" P-values to your question. Perhaps you will get more satisfactory results if you make only the very few most important ad hoc comparisons. You seem to be paying a very heavy Bonferroni adjustment penalty for making so many.

$\endgroup$
7
  • $\begingroup$ could you please check the question, I have included the average ranks and the adjusted P-values. $\endgroup$
    – SHdotCom
    Sep 30, 2021 at 17:36
  • $\begingroup$ Saw them. See my corresponding Addendum. Maybe confine comparisons to the most important among Methods 3, 6. 7. and 8. $\endgroup$
    – BruceET
    Sep 30, 2021 at 17:42
  • $\begingroup$ could you please check the Addition information. If this result is correct, what we can conclude from the statistical analysis? Are these methods have the same performance or different, and how to determine which method is better? $\endgroup$
    – SHdotCom
    Oct 1, 2021 at 6:52
  • $\begingroup$ I have included the original data (at the top of the question). $\endgroup$
    – SHdotCom
    Oct 1, 2021 at 14:54
  • 1
    $\begingroup$ If you found this answer helpful, then please consider upvoting and/or accepting it. $\endgroup$ Oct 9, 2021 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.