This post does not include an answer for my question.
I have perform a Friedman test on a results obtained by several methods on multiple datasets, as described in this paper: "Statistical Comparisons of Classifiers over Multiple Data Sets"
The Friedman's statistic is (16.667) which exceeds the Chi-Square critical value (14.067) so the null hypothesis H0 (no difference between the methods) must be rejected. However, when I used a post-hoc test (Shaffer test) to identify the differences, All the adjusted P-values obtained by Shaffer test are bigger than level of significance 0.05. This means that there are no statically significant differences between the results obtained by these methods.
Is this result correct or not? Why the H0 is rejected at the first stage, then not rejected at the post-hoc test?
If this result is correct, what does it mean?
Data sets,Method1, Method2, Method3, Method4, Method5, Method6, Method7, Method8 F1,1.3708E-01,1.0227E-01,1.2801E-01,1.0512E-01,1.4476E-01,1.1309E-01,1.4002E-01,1.2801E-01 F2,2.2731E-01,2.2758E-01,2.1174E-01,2.2516E-01,2.1865E-01,2.1740E-01,2.2620E-01,2.1174E-01 F3,8.6773E-02,8.5749E-02,8.1918E-02,8.5238E-02,8.7778E-02,8.3436E-02,8.9080E-02,8.1918E-02 F4,1.7313E-01,1.7626E-01,1.6378E-01,1.8294E-01,2.0685E-01,8.0293E-02,2.0000E-01,1.6378E-01
Method1 3.25 Method2 4.25 Method3 6.5 Method4 4.75 Method5 2.25 Method6 6.5 Method7 2 Method8 6.5
0.262494 0.262494 0.262494 0.296897 0.296897 0.296897 1.272641 1.272641 1.272641 1.797619 2.382635 3.102894 3.102894 3.102894 3.102894 3.22677 3.747857 3.747857 3.747857 3.747857 3.763891 3.94592 3.94592 3.94592 3.94592 3.94592 3.94592 3.94592
According to that paper, "If the null-hypothesis is rejected, we can proceed with a post-hoc test." A a post-hoc test is used to identify which pairs of algorithms are significantly different than each other. Thus, in a multiple comparison, all possible pairwise comparisons need to be computed (N × N comparison). The number of possible comparisons in an all pairwise comparisons is N(N-1)/2. So we have 8(8 - 1)/2=28 hypotheses and an adjusted P-Value for each hypothes.