I have k classifiers and n datasets, and I have only one accuracy measurement (which is actually the average of three independent repetitions of the 5-fold-CV, i.e. average over 15 accuracy values) for each algorithm+dataset combination.

I have read Demsar 2006 and some more recent papers. I first use an Aligned Friedman test and, in case the result is significant, I apply a post-hoc test. The problem is, for some specific reasons, I do not just want to compare a control method vs the rest as usual (i.e., k-1 comparisons; a lot of post-hoc for 1 vs All exist with p-value correction for k-1 comparisons) but, instead, I want to perform all vs all comparisons.

I know I could use multiple paired Wilcoxon tests ("Wilcoxon signed rank test") with some p-value correction (Holm, etc), just as the pairwise.wilcox.test function in R, which automatically corrects the unadjusted Wilcoxon p-values to account for all comparisons. But I suspect there are more appropriate post-hoc tests after Aligned Friedman, could you give me advice please?

Further, I know this commonly used post hoc for 1 vs all after Aligned Friedman:

$z_{ij} = (R_i - R_j)/\sqrt{\frac{k(kn+1)}{6}}$

where $R_i$ and $R_j$ are Aligned Friedman average ranks of algorithms i-th (control) and j-th. Note $z_{ij}$ = $-z_{ji}$. Since z follows a N(0,1), the unadjusted p-values of $z_{ij}$ and $z_{ji}$ are the same, pval = 2*pnorm(-abs(z)).

My second question is: Is it correct to use z to perform all vs all (for k=4, I would do 6 comparisons), providing that every pair of algorithms is compared just once, and that I perform p-value adjustment (via Holm for instance) of the 6 unadjusted pvalues calculated with the above formula? Or the z in this post-hoc was specifically conceived for 1 vs all after Aligned Friedman and cannot be used for anything else?

Thanks a lot


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.