2
$\begingroup$

I'm looking to run a bunch of t-tests, and I'm trying to figure out the appropriate time to apply an FDR correction.

I have four conditions and am doing pairwise comparisons amongst these conditions, so I have six pairwise comparisons/t-tests to run on each variable. I have 30 variables.

Does it make more sense to:

  • Do all of the pairwise t-tests, take all of the p-values, and perform one FDR correction? I.e.: I have 6 p-values (6 pairwise comparisons) for the 30 variables and run one FDR correction on the 180 p-values?

  • Or to perform an FDR correction for each pairwise t-test for each variable? I.e.: Run an FDR correction for the 6 p-values for each variable, for a total of 30 FDR corrections.

ETA:

To clarify, my variables are not dependent (my title was edited)--perhaps 'variables' is not the correct term. I am looking at changes in gene expression for 30 different unrelated genes. The conditions are the same for each gene though. So for gene 1, I am running pairwise t-tests for condition A vs B, A vs C, A vs D, B vs C, B vs D, and C vs D, and then repeating this for each gene.

I've just noticed this question, in which the asker has a similar set-up to me: Multiple testing and FDR on multiple-pairs

I hadn't thought about running FDR for each pairwise comparison (6 FDR corrections for my case), though I'm not sure this would be the best option. Advice?

$\endgroup$
1
$\begingroup$

Benjamini-Hochberg procedure (FDR) generally assumes tests are independent from each other (but see this), so it may not be an ideal test for among-conditions assessment (the 6 pairwise comparisons). Otherwise, 'to run a bunch of tests' sounds like a single-family/question approach so you'd need to apply correction to the entire population of p-values generated. In other words, you just set your bet 180 times, chances you 'win' in this roulette are quite a bit higher than if you were to set 1 bet. A more cohesive design e.g. a two-way ANOVA with Tukey-Kramer post-hoc comparisons, assuming the 4 conditions are the same throughout, would be much preferred.

$\endgroup$
  • $\begingroup$ BH does allow for certain types of dependence! $\endgroup$ – Christoph Hanck Jul 30 '15 at 19:11
  • 1
    $\begingroup$ @ChristophHanck - ok, clarified $\endgroup$ – katya Jul 30 '15 at 19:31
  • $\begingroup$ To clarify, my variables are not dependent (my title was edited)--perhaps 'variables' is not the correct term. I am looking at changes in gene expression for 30 different unrelated genes. The conditions are the same for each gene though. So for gene 1, I am running pairwise t-tests for condition A vs B, A vs C, A vs D, B vs C, B vs D, and C vs D, and then repeating this for each gene. $\endgroup$ – hmg Jul 31 '15 at 14:30
  • $\begingroup$ Note also the paper: Joseph P. Romano · Azeem M. Shaikh · Michael Wolf: "Control of the false discovery rate under dependence using the bootstrap and subsampling." Test (2008) 17: 417–442. The simulation results that are reported there suggest that BH is effective under quite general correlation structures. Not a proof, but BH seems clearly hard to break $\endgroup$ – John Maindonald Oct 1 '17 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.