4
$\begingroup$

This feels like a really basic question, but I haven't found a definitive answer.

When correcting p-values for multiple comparisons, are you supposed to only run the analysis/correction on the p-values that are significant before correction, or do you run it on all of them?

Or does it depend on whether you are running a FWER (Bonferroni, Holm, etc.) or a FDR (Benjamini-Hochberg, etc.) correction?

$\endgroup$
6
  • $\begingroup$ Do you really need to apply any correction to your p values? I've seen hundreds of publications with up to 100 tests with no correction for multiple testing; it all depends on your research design and question. But in general, if you need to adjust for multiple testing you apply that to all p values a priori. $\endgroup$ – Adam Robinsson Jan 3 '15 at 17:37
  • 2
    $\begingroup$ Correcting p-values makes them but higher. So, nonsignificant p-values will never become more significant. So, you may "correct" for yourself only the ones formerly significant, to see. But the amount of correction is dependent on the number of tests in your "family of tests", not the number of tests appearing "signifinant" in it. $\endgroup$ – ttnphns Jan 3 '15 at 17:41
  • $\begingroup$ @ttnphns, I understand that correcting makes them larger, so non-significant ones won't become significant. I'm asking more about the significant ones becoming non-significant, because the number of tests affects that. $\endgroup$ – Anthony Jan 3 '15 at 17:54
  • $\begingroup$ For me the cause for confusion is mainly that FDR correction controls incorrectly rejected null hypotheses... so I wonder if I should only be inputting the rejected null hypotheses (i.e., significant p-values). $\endgroup$ – Anthony Jan 3 '15 at 18:02
  • $\begingroup$ @ttnphns I don't think this is the "advocated" approach, unless you refuse to present any $p$-values at all and instead describe findings as significant or not. At least if you corrected select findings, you must use some kind of notation to separate the non-significant, uncorrected $p$-values from the significant, corrected ones. Otherwise the reader is led to believe many more analyses might have been borderline significant and would otherwise have been significant if multiplicity had not been accounted for. The $p$-value is NOT a distinct threshold between significance and not. $\endgroup$ – AdamO Jul 19 '16 at 15:21
4
$\begingroup$

A general bonferroni correction requires that you correct based on the total number of tests that have been run. Given that corrections for multiple comparisons should be planned for in advance of the actual analysis, the results of the tests in question have no bearing on the validity of the multiple comparisons correction method.

Consider the case where you run 100 random tests, and you find that only one of them gives you significant results. If you only 'corrected' for that one test, then that one test remains significant even though you ran 100 tests, clearly this wouldn't make sense.

There are of course other ways to correct for random comparisons, especially if the tests are related in some form. However, all of the methods that I know of are independent of the actual result of the tests.

$\endgroup$
3
  • $\begingroup$ That makes sense for Bonferroni and other FWER corrections. What about FDR which focuses specifically on the rejected null hypotheses (i.e., significant p-values)? $\endgroup$ – Anthony Jan 3 '15 at 21:12
  • 1
    $\begingroup$ Indeed it does matter and using FDR methods does only take into account the significant p-values. FDR methods are generally designed for when you have a very large number of tests, and the Bonferonni adjustment makes it essentially impossible to find any significant differences. For FDR you specify a new q-value threshold which roughly represents the proportion of significant test results you are willing to declare as false positives. $\endgroup$ – Mensen Jan 5 '15 at 10:39
  • $\begingroup$ Given that this proportion is also fairly low, you need to have a large number of significant results for this small proportion to be meaningful (e.g. at least 10 significant results for a q-value threshold of 0.1 to lead you to rejection a single null hypothesis). FDR is less conservative than FWER methods, and it will highly depend on your application as to whether someone believes your results. Exploratory studies may accept FDR, clinical trials would probably not. $\endgroup$ – Mensen Jan 5 '15 at 10:41
1
$\begingroup$

I have often interpretted a 0.06 $p$-value as "borderline" significant, especially in small sample size studies. If you are presenting multiple $p$-values for an analysis, I would strongly advocate presenting all $p$-values on the same scale.

That means either corrected or not, but not a mixture thereof. This is misleading the reader and begs the question why one would even present $p$-values at all.

If you use a FDR approach, the raw $p$-values must be presented. In permutation testing or bonferroni you may either transform all the $p$-values onto a non-uniform scale with a family wise error rate of $\alpha$ or present the untransformed $p$-values and annotate which ones are actually significant at the $\alpha$ level, (so a 0.02 finding may not be significant in the family wise test).

$\endgroup$
0
$\begingroup$

Mensen's commment is incorrect. Whether controlling FWER or merely FDR, your corrections must account for ALL tests conducted. If you only input the p-values that are < .05, an FDR-controlling procedure will invariably find all of them significant at the .05 level (making FDR control pointless).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.