I'm confused as to when adjustments for multiple tests need to be used. I'm an undergrad who's familiarising himself with stats, so apologies if my question sounds dumb, but I haven't found any satisfactory answer myself. I understand the purpose of these corrections as well as that there are different approaches (controlling FWER vs. FDR). That being said, I have found many people not using any such corrections except for specific cases such as multiple comparisons in ANOVA. One of our lecturers said that you're adjusting when the tests are not independent. Others have told me that it doesn't really matter and as long as you're following NHST, you should control for all your tests performed on one sample (seems kinda strict). Some say that you should decide based on what your "family of tests" is. And also, others seem to highlight the importance of formulating the hypotheses a priori, in which case you don't always need to adjust. Is there some link between all of these views that I am missing?

Also, suppose I'm using ANOVA with 2 categorical predictors and an interaction. Why don't I need to adjust my F-test p-values for each of the predictors and the interaction? Also, when we use multiple comparisons or contrasts, why do we adjust taking into account only these tests and not including the p-values from the F-tests?

  • $\begingroup$ For your ANOVA idea, what exactly would you want to test? It is perfectly acceptable to test one regression coefficient and not adjust for the fact that you have 3 other parameters, though you wouldn’t report other coefficients as significant, even if their p-values are less than $\alpha$. $\endgroup$ – Dave Apr 30 at 15:35
  • $\begingroup$ I am asking about the general principle of the corrections, so don't think too specifically about the example I'm gonna give now. Say I want to test if well-being (measured on a scale) is influenced by pet ownership and marital status (both as categorical variables). First, lots of people would start with F-tests, one for each predictor (including the interaction). Then, either contrasts with t-tests or post-hoc tests follow - and usually, only those are corrected for multiple comparisons. However, I used three extra tests (the F-tests) before those, shouldn't people correct for those as well? $\endgroup$ – danjeff Apr 30 at 15:55
  • $\begingroup$ Okay, you’re doing all of them. That’s different from most of what I do where I’m interested in the treatments and account for covariates/factors but don’t test them explicitly. Yes, if you do $k$-many tests, you should adjust for $k$-many tests. (I don’t mean in your entire career or in the entire history of hypothesis testing.) $\endgroup$ – Dave Apr 30 at 16:21

Is there any consensus on adjusting p-values for multiple tests?

Not really.

This really depends on your field, your target audience, your question, and the opinion of the individual. There isn't one and only one accepted way of doing this and I've had conflicting opinions from reviewers from international peer-reviewed journal submissions, with being told the Bonferonni corrections is both necessary and overly conservative and we should use a FDR. Bonferonni / FWER type corrections tend to be better understood by non-statisticians though, which is sometimes relevant depending on your target audience and the actual purpose of your work.

Edit: A comment below seems to indicate this answer implied whether to correct is subject to opinion. The answer was never supposed to imply that, only that which specific control method you use is often subject to field / audience / question, and that FWER is more widely understood.

In general terms: FDR is more powerful than FWER but does allow for some false positives. I generally prefer FWER, because if I find a borderline significant p value and report it negative, someone else might repeat the study, meta-analyse our results, and find a positive result. But if I find a false positive, it's less likely to be checked, and even it is and found to be negative by someone else with bigger numbers, positive findings propagate faster than negative findings.

| cite | improve this answer | |
  • 1
    $\begingroup$ I grant there are some fuzzy areas here. But statistical expertise must be considered ahead of broad opinion. // At least among competent applied statisticians, there is agreement about the need to control family error rates to avoid false discovery. Various statisticians have their favorite methods, but mainly to achieve the same ends. // You can get 'conflicting opinions' as to whether the earth is flat. Whether to use such methods at all ought not to be left up to a vote of journal editors, researchers, target audiences, etc. in various non-statistical fields. $\endgroup$ – BruceET Apr 29 at 23:58
  • $\begingroup$ I agree that some sort of multiple detection correcting should always be used. I'm a mathematician working in the medical field and often see clinicians present 20 statistical tests and then say 'hey look this one had p=0.04' and I'm there with my face in my hands. But as for which methods to use, I generally prefer FWER over FDR because 1) false positives are more dangerous than false negatives (which can create true positives in meta analyses) and 2) if I present FDR they look at me like I'm an alien from a distant land and the actual message gets lost in the discussion of FDR vs FWER $\endgroup$ – H. Green Apr 30 at 10:05
  • $\begingroup$ Thank you for your answer! As you're saying, I do realise that the choice of a particular method depends on the overall context (ofc not completely). I think my question is more related to the actual control of the error rates. In my example using ANOVA, does the fact that I am using 4 F-tests in the output not inflate the FWER/FDR? Because if so, then why should't researchers control for those? I understand that using these can inflate the type II error, but in that case, isn't it "objectively better" to adjust and do an apriori power analysis for that to ensure it has sufficient power? $\endgroup$ – danjeff Apr 30 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.