tl;dr: I feel like multiple comparison can be easily manipulated to have significant results.

As far as I understand, the concept of multiple comparison problem is very simple. As we are conducting every tests with alpha level 0.05, the false positive rates will increase dramatically as the number of tests conducted increases.

Here comes my confusion, while I was reading researchers correcting for multiple comparison (say for instance you are doing correlations between 5 behavioral measures and 10 brain signals from different location, therefore 50 correlations in total), no matter you choose FDR, bonferroni or others, you could possibly find none of the results is significant due to that large amount of tests. But one can always "claim" he/she only did correlation tests, for example, just on 1 behavioral measure, and those insignificant results then become significant.

I have been discussing with my colleagues but we dont have a good explanation on this confusion, it made me hard to believe "significant" results from papers. Of course correction is generally recommended, but who know if these researchers were not telling lies? Claim that they were doing 5 tests but actaully they have done 100 tests?

  • 3
    $\begingroup$ The assumption is that statisticians follow a code of ethics which requires telling the truth about the design and analysis. Some detective work might show when the truth has not been told, but there is no guarantee of that. // One does not do multiple comparison procedures for an effect unless the F-test for the effect is signif at the stated level $\alpha$ (often 5%). If the F-test shows significant diff's among the levels of the factor, then many statisticians would feel comfortable saying that the largest diff among them is signif. even if mult comp proc doesn't declare it signif. $\endgroup$ – BruceET Dec 28 '18 at 9:45
  • 1
    $\begingroup$ In addition, a prespecified statistical plan is used in most high quality research projects, and thus selective reporting of multiple tests is discouraged or unfeasible. $\endgroup$ – Joe_74 Dec 28 '18 at 11:31
  • $\begingroup$ @BruceET How should I understand the largest diff is significant? Say if multiple tests were significant before correcting but not after, how should we tell others about such results and most importantly, convince them this is a really important result? $\endgroup$ – matthewhang Dec 29 '18 at 5:19
  • $\begingroup$ @Joe_74 Yea i thought it would be more like a ethical problem. This multiple correction concept is weird to me when I just know about it (i wasnt major in stat). As much of the reviewers look at the p-value, it is not hard to imagine the existence of selective reporting... $\endgroup$ – matthewhang Dec 29 '18 at 5:19
  • $\begingroup$ If you have a prespecified protocol and a primary endpoint with explicit analysis, then there is no ethical issue. You have problems only if you have a non-significant primary endpoint but borderline secondary ones. The best thing in that scenario is to consider most results hypothesis generating and rely on (subsequently) meta-analysis. $\endgroup$ – Joe_74 Dec 30 '18 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.