Suppose we have algorithms $A_k$ and $B_k$, $1\leq k\leq n$, and $\alpha = 0.05$.
Friedman test for the algorithms $A_k$ shows that the null hypothesis that all $A_k$s perform equally well, can be rejected. Suppose $p_A = 10^{-5}\ll\alpha$.
Friedman test for the algorithms $B_k$ shows that the null hypothesis that all $B_k$s perform equally well, cannot be rejected. Suppose $p_B = 0.4$.
My problem is that the $p$-value $p_{AB}$ from the Friedman test for $A_k$s and $B_k$s is greater than $\alpha$, hence I cannot apply Nemenyi post-hoc test if I blindly follow the procedure
- perform Friedman on algorithms
- if $H_0$ is rejected, proceed to Nemenyi on the same algorithms
Question: Are the results from Nemenyi test for all algorithms still valid?
I am not sure, but I would say yes because
- I have shown that already not all $A_k$s have the same performance (and the results would still be significant with $2p_A$, i.e., if I used Bonferroni correction for multiple-hypothesis testing)
- Nemenyi for $A_k$s and $B_k$s found two groups of algorithms, i.e., some differences.
