When to correct p-values in multiple comparisons?

Question

I'm afraid that related questions didn't answer mine. We evaluate the performances of >2 classifiers (machine learning). Our Null hypothesis is that performances do not differ. We perform parametric (ANOVA) and non-parametric (Friedman) tests to evaluate this hypothesis. If they're significant, we want to find out which classifiers differ in a post-hoc quest.

My question is twofold:

1) Is a correction of p-values after multiple comparisons testing necessary at all? The German Wikipedia site on "Alphafehler Kumulierung" says that the problem only occurs if multiple hypotheses are tested on the same data. When comparing classifiers (1,2),(1,3),(2,3), data only partially overlaps. Is it still required to correct the p-values?

2) P-value correction is often used after pairwise testing with a t-test. Is it also necessary when doing specialised post-hoc tests, such as Nemenyi's (non-parametric) or Tukey's HSD test? This answer says "no" for Tukey's HSD: Does the Tukey HSD test correct for multiple comparisons?. Is there a rule or do I have to look this up for every potential post-hoc test?

Thanks!

Answer 1

Answer to question 1
You need to adjust for multiple comparisons if you care about the probability at which you will make a Type I error. A simple combination of metaphor/thought experiment may help:

Imagine that you want to win the lottery. This lottery, strangely enough, gives you a 0.05 chance of winning (i.e. 1 in 20). M is the cost of the ticket in this lottery, meaning that your expected return for a single lottery call is M/20. Now even stranger, imagine that for unknown reasons, this cost, M, allows you to have as many lottery tickets as you want (or at least more than two). Thinking to yourself "the more you play, the more you win" you grab a bunch of tickets. Your expected return on a lottery call is no longer M/20, but something a fair bit larger. Now replace "winning the lottery" with "making a Type I error."

If you do not care about errors, and you don't care about people repeatedly and mockingly directing your attention to a certain cartoon about jellybeans, then go ahead and do not adjust for multiple comparisons.

The "same data" issue arises in family-wise error correction methods (e.g. Bonferroni, Holm-Sidák, etc.), since the concept of "family" is somewhat vague. However, the false discovery rate methods (e.g. Benjamini and Hochberg, Benjamini and Yeuketeli, etc.) have a property that their results are robust across different groups of inferences.

Answer to question 2
Most pairwise tests require correction, although there are stylistic and disciplinary differences in what gets called a test. For example, some folks refer to "Bonferroni t tests" (which is a neat trick, since Bonferroni developed neither the t test, nor the Bonferroni adjustment for multiple comparisons :). I personally find this dissatisfying, as (1) I would like to make a distinction between conducting a group of statistical tests, and adjusting for multiple comparisons in order to effectively understand the inferences I am making, and (2) when someone comes along with a new pairwise test founded on a solid definition of $\alpha$, then I know I can perform adjustments for multiple comparisons.