Friedman's test is very significant, but its post hoc comparisons are not significant

Question

This post does not include an answer for my question.

I have perform a Friedman test on a results obtained by several methods on multiple datasets, as described in this paper: "Statistical Comparisons of Classifiers over Multiple Data Sets"

The Friedman's statistic is (16.667) which exceeds the Chi-Square critical value (14.067) so the null hypothesis H0 (no difference between the methods) must be rejected. However, when I used a post-hoc test (Shaffer test) to identify the differences, All the adjusted P-values obtained by Shaffer test are bigger than level of significance 0.05. This means that there are no statically significant differences between the results obtained by these methods.

Is this result correct or not? Why the H0 is rejected at the first stage, then not rejected at the post-hoc test?

If this result is correct, what does it mean?

Original Data:

    Data sets,Method1, Method2, Method3, Method4, Method5, Method6, Method7, Method8
    F1,1.3708E-01,1.0227E-01,1.2801E-01,1.0512E-01,1.4476E-01,1.1309E-01,1.4002E-01,1.2801E-01
    F2,2.2731E-01,2.2758E-01,2.1174E-01,2.2516E-01,2.1865E-01,2.1740E-01,2.2620E-01,2.1174E-01
    F3,8.6773E-02,8.5749E-02,8.1918E-02,8.5238E-02,8.7778E-02,8.3436E-02,8.9080E-02,8.1918E-02
    F4,1.7313E-01,1.7626E-01,1.6378E-01,1.8294E-01,2.0685E-01,8.0293E-02,2.0000E-01,1.6378E-01

Average Ranks:

Method1  3.25
Method2  4.25
Method3  6.5
Method4  4.75
Method5  2.25
Method6  6.5
Method7  2
Method8  6.5

Adjusted P-values:

0.262494
0.262494
0.262494
0.296897
0.296897
0.296897
1.272641
1.272641
1.272641
1.797619
2.382635
3.102894
3.102894
3.102894
3.102894
3.22677
3.747857
3.747857
3.747857
3.747857
3.763891
3.94592
3.94592
3.94592
3.94592
3.94592
3.94592
3.94592

Addition information

According to that paper, "If the null-hypothesis is rejected, we can proceed with a post-hoc test." A a post-hoc test is used to identify which pairs of algorithms are significantly different than each other. Thus, in a multiple comparison, all possible pairwise comparisons need to be computed (N × N comparison). The number of possible comparisons in an all pairwise comparisons is N(N-1)/2. So we have 8(8 - 1)/2=28 hypotheses and an adjusted P-Value for each hypothes.

Answer 1

Friedman's test and the ad hoc test use somewhat different distributions and criteria. In any case, Bonferroni ad hoc tests can be too conservative (not likely enough to reject), especially if you are making several ad hoc comparisons.

You do not give the P-values, the number of levels, level means/medians for the Friedman test, or ad hoc adjusted P-values. So it is not possible to give more than a guess at he particulars, trying to answer your question.

Here is my guess: It looks as if the main Friedman test may have been significant at something like the 2% significance level with maybe 8 levels of the main factor, and the most nearly significant ad hoc test may have been significant at something like the 7% level (adjusted). If so, the results may be annoying, but not wrong--or even entirely unexpected.

qchisq(.95, 7)
[1] 14.06714
1 - pchisq(16.67, 7)
[1] 0.01965259

If the overall Friedman test is significant, then it is reasonable to guess that the two levels with the greatest difference in location must be significantly different, but you may not be able to test for sure how locations of the (8 or so) levels are ranked.

In doing a 'power and sample size' analysis, in advance of taking data, one should try to determine how many 'blocks' are needed for ad hoc tests with a desired effect size to be detectable. This is not always easy to do.

Addendum. I see that you have appended a couple of dozen "adjusted" P-values to your question. Perhaps you will get more satisfactory results if you make only the very few most important ad hoc comparisons. You seem to be paying a very heavy Bonferroni adjustment penalty for making so many.