I'm afraid that related questions didn't answer mine. We evaluate the performances of >2 classifiers (machine learning). Our Null hypothesis is that performances do not differ. We perform parametric (ANOVA) and non-parametric (Friedman) tests to evaluate this hypothesis. If they're significant, we want to find out which classifiers differ in a post-hoc quest.
My question is twofold:
1) Is a correction of p-values after multiple comparisons testing necessary at all? The German Wikipedia site on "Alphafehler Kumulierung" says that the problem only occurs if multiple hypotheses are tested on the same data. When comparing classifiers (1,2),(1,3),(2,3), data only partially overlaps. Is it still required to correct the p-values?
2) P-value correction is often used after pairwise testing with a t-test. Is it also necessary when doing specialised post-hoc tests, such as Nemenyi's (non-parametric) or Tukey's HSD test? This answer says "no" for Tukey's HSD: Does the Tukey HSD test correct for multiple comparisons?. Is there a rule or do I have to look this up for every potential post-hoc test?