4
$\begingroup$

This feels like a really basic question, but I haven't found a definitive answer.

When correcting p-values for multiple comparisons, are you supposed to only run the analysis/correction on the p-values that are significant before correction, or do you run it on all of them?

Or does it depend on whether you are running a FWER (Bonferroni, Holm, etc.) or a FDR (Benjamini-Hochberg, etc.) correction?

Improve this question
$\endgroup$
6
  • $\begingroup$ Do you really need to apply any correction to your p values? I've seen hundreds of publications with up to 100 tests with no correction for multiple testing; it all depends on your research design and question. But in general, if you need to adjust for multiple testing you apply that to all p values a priori. $\endgroup$
    – Adam Robinsson
    Jan 3, 2015 at 17:37
  • 2
    $\begingroup$ Correcting p-values makes them but higher. So, nonsignificant p-values will never become more significant. So, you may "correct" for yourself only the ones formerly significant, to see. But the amount of correction is dependent on the number of tests in your "family of tests", not the number of tests appearing "signifinant" in it. $\endgroup$
    – ttnphns
    Jan 3, 2015 at 17:41
  • $\begingroup$ @ttnphns, I understand that correcting makes them larger, so non-significant ones won't become significant. I'm asking more about the significant ones becoming non-significant, because the number of tests affects that. $\endgroup$
    – Anthony
    Jan 3, 2015 at 17:54
  • $\begingroup$ For me the cause for confusion is mainly that FDR correction controls incorrectly rejected null hypotheses... so I wonder if I should only be inputting the rejected null hypotheses (i.e., significant p-values). $\endgroup$
    – Anthony
    Jan 3, 2015 at 18:02
  • $\begingroup$ @ttnphns I don't think this is the "advocated" approach, unless you refuse to present any $p$-values at all and instead describe findings as significant or not. At least if you corrected select findings, you must use some kind of notation to separate the non-significant, uncorrected $p$-values from the significant, corrected ones. Otherwise the reader is led to believe many more analyses might have been borderline significant and would otherwise have been significant if multiplicity had not been accounted for. The $p$-value is NOT a distinct threshold between significance and not. $\endgroup$
    – AdamO
    Jul 19, 2016 at 15:21

3 Answers 3

Reset to default
4
$\begingroup$

A general bonferroni correction requires that you correct based on the total number of tests that have been run. Given that corrections for multiple comparisons should be planned for in advance of the actual analysis, the results of the tests in question have no bearing on the validity of the multiple comparisons correction method.

Consider the case where you run 100 random tests, and you find that only one of them gives you significant results. If you only 'corrected' for that one test, then that one test remains significant even though you ran 100 tests, clearly this wouldn't make sense.

There are of course other ways to correct for random comparisons, especially if the tests are related in some form. However, all of the methods that I know of are independent of the actual result of the tests.

Improve this answer
$\endgroup$
3
  • $\begingroup$ That makes sense for Bonferroni and other FWER corrections. What about FDR which focuses specifically on the rejected null hypotheses (i.e., significant p-values)? $\endgroup$
    – Anthony
    Jan 3, 2015 at 21:12
  • 1
    $\begingroup$ Indeed it does matter and using FDR methods does only take into account the significant p-values. FDR methods are generally designed for when you have a very large number of tests, and the Bonferonni adjustment makes it essentially impossible to find any significant differences. For FDR you specify a new q-value threshold which roughly represents the proportion of significant test results you are willing to declare as false positives. $\endgroup$
    – Mensen
    Jan 5, 2015 at 10:39
  • $\begingroup$ Given that this proportion is also fairly low, you need to have a large number of significant results for this small proportion to be meaningful (e.g. at least 10 significant results for a q-value threshold of 0.1 to lead you to rejection a single null hypothesis). FDR is less conservative than FWER methods, and it will highly depend on your application as to whether someone believes your results. Exploratory studies may accept FDR, clinical trials would probably not. $\endgroup$
    – Mensen
    Jan 5, 2015 at 10:41
1
$\begingroup$

I have often interpretted a 0.06 $p$-value as "borderline" significant, especially in small sample size studies. If you are presenting multiple $p$-values for an analysis, I would strongly advocate presenting all $p$-values on the same scale.

That means either corrected or not, but not a mixture thereof. This is misleading the reader and begs the question why one would even present $p$-values at all.

If you use a FDR approach, the raw $p$-values must be presented. In permutation testing or bonferroni you may either transform all the $p$-values onto a non-uniform scale with a family wise error rate of $\alpha$ or present the untransformed $p$-values and annotate which ones are actually significant at the $\alpha$ level, (so a 0.02 finding may not be significant in the family wise test).

Improve this answer
$\endgroup$
0
$\begingroup$

Mensen's commment is incorrect. Whether controlling FWER or merely FDR, your corrections must account for ALL tests conducted. If you only input the p-values that are < .05, an FDR-controlling procedure will invariably find all of them significant at the .05 level (making FDR control pointless).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.