11
$\begingroup$

I'm afraid that related questions didn't answer mine. We evaluate the performances of >2 classifiers (machine learning). Our Null hypothesis is that performances do not differ. We perform parametric (ANOVA) and non-parametric (Friedman) tests to evaluate this hypothesis. If they're significant, we want to find out which classifiers differ in a post-hoc quest.

My question is twofold:

1) Is a correction of p-values after multiple comparisons testing necessary at all? The German Wikipedia site on "Alphafehler Kumulierung" says that the problem only occurs if multiple hypotheses are tested on the same data. When comparing classifiers (1,2),(1,3),(2,3), data only partially overlaps. Is it still required to correct the p-values?

2) P-value correction is often used after pairwise testing with a t-test. Is it also necessary when doing specialised post-hoc tests, such as Nemenyi's (non-parametric) or Tukey's HSD test? This answer says "no" for Tukey's HSD: Does the Tukey HSD test correct for multiple comparisons?. Is there a rule or do I have to look this up for every potential post-hoc test?

Thanks!

$\endgroup$
  • $\begingroup$ Why are you performing both ANOVA and Friedman tests? $\endgroup$ – Alexis Jul 22 '14 at 17:51
  • $\begingroup$ It's about an automated testing framework which should provide the reviewer with both a parametric and non-parametric alternative, if the parametric assumptions aren't met. $\endgroup$ – Chris Jul 22 '14 at 21:03
  • 1
    $\begingroup$ About the omnibus tests you mentioned: (A) if your data groups are independent, you should use either ANOVA (parametric) or Kruskal-Wallis (non-parametric) test; (B) if your groups are dependent (e.g., repeated measures) then you should use either repeated measures ANOVA (parametric) or Friedman (non-parametric) test. (Classical) ANOVA and Friedman test as its alternative does not sound correct. $\endgroup$ – GegznaV Nov 26 '17 at 11:59
10
$\begingroup$

Answer to question 1
You need to adjust for multiple comparisons if you care about the probability at which you will make a Type I error. A simple combination of metaphor/thought experiment may help:

Imagine that you want to win the lottery. This lottery, strangely enough, gives you a 0.05 chance of winning (i.e. 1 in 20). M is the cost of the ticket in this lottery, meaning that your expected return for a single lottery call is M/20. Now even stranger, imagine that for unknown reasons, this cost, M, allows you to have as many lottery tickets as you want (or at least more than two). Thinking to yourself "the more you play, the more you win" you grab a bunch of tickets. Your expected return on a lottery call is no longer M/20, but something a fair bit larger. Now replace "winning the lottery" with "making a Type I error."

If you do not care about errors, and you don't care about people repeatedly and mockingly directing your attention to a certain cartoon about jellybeans, then go ahead and do not adjust for multiple comparisons.

The "same data" issue arises in family-wise error correction methods (e.g. Bonferroni, Holm-Sidák, etc.), since the concept of "family" is somewhat vague. However, the false discovery rate methods (e.g. Benjamini and Hochberg, Benjamini and Yeuketeli, etc.) have a property that their results are robust across different groups of inferences.


Answer to question 2
Most pairwise tests require correction, although there are stylistic and disciplinary differences in what gets called a test. For example, some folks refer to "Bonferroni t tests" (which is a neat trick, since Bonferroni developed neither the t test, nor the Bonferroni adjustment for multiple comparisons :). I personally find this dissatisfying, as (1) I would like to make a distinction between conducting a group of statistical tests, and adjusting for multiple comparisons in order to effectively understand the inferences I am making, and (2) when someone comes along with a new pairwise test founded on a solid definition of $\alpha$, then I know I can perform adjustments for multiple comparisons.

$\endgroup$
  • 2
    $\begingroup$ +1 for a comprehensive and humorous answer (and for referring to xkcd). In particular, you also tackled my yet unverbalized question whether there's a difference between "Bonferroni-test" and "Bonferroni-correction". Nevertheless, would you mind to explain the multiple comparisons problem in terms of my problem description? I understand that one classifiers is like a treatment group with no/blue/green/... jelly beans in the comic. $\endgroup$ – Chris Jul 23 '14 at 7:15
  • $\begingroup$ @Chris You are welcome... I am not quite sure what you are asking. Yes multiple comparisons is needed. Yes, you can perform FWER or FDR adjustments on any pairwise test that returns $p$-values (the procedures generally modify $p$-values, or modify the rejection level, either overall, or sequentially). $\endgroup$ – Alexis Jul 24 '14 at 3:40
  • $\begingroup$ I think that's fine, thank you so much! It might take me some more time to apply the lottery example to my use-case, but I got the idea. $\endgroup$ – Chris Jul 24 '14 at 7:17
  • $\begingroup$ @Chris understand that the lottery was just a metaphor. If you need help applying FWER or FDR methods, check out the Wikipedia entries, search for related questions here, or, perhaps, ask a new question about that. :) $\endgroup$ – Alexis Jul 24 '14 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.