1
$\begingroup$

Suppose we have algorithms $A_k$ and $B_k$, $1\leq k\leq n$, and $\alpha = 0.05$.

Friedman test for the algorithms $A_k$ shows that the null hypothesis that all $A_k$s perform equally well, can be rejected. Suppose $p_A = 10^{-5}\ll\alpha$.

Friedman test for the algorithms $B_k$ shows that the null hypothesis that all $B_k$s perform equally well, cannot be rejected. Suppose $p_B = 0.4$.

My problem is that the $p$-value $p_{AB}$ from the Friedman test for $A_k$s and $B_k$s is greater than $\alpha$, hence I cannot apply Nemenyi post-hoc test if I blindly follow the procedure

  • perform Friedman on algorithms
  • if $H_0$ is rejected, proceed to Nemenyi on the same algorithms

Question: Are the results from Nemenyi test for all algorithms still valid?

I am not sure, but I would say yes because

  • I have shown that already not all $A_k$s have the same performance (and the results would still be significant with $2p_A$, i.e., if I used Bonferroni correction for multiple-hypothesis testing)
  • Nemenyi for $A_k$s and $B_k$s found two groups of algorithms, i.e., some differences.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.