I want to discover which genes are expressed in only one of five treatments. This is my pipeline:

  1. ANOVA between the five treatments
  2. Holm multiple testing correction
  3. Tukey for significant genes discovered in step 2

My question is: Should I also correct Tukey p-values for example multiplying the p-value by the number of significant ANOVA p-values (Bonferroni correction on the number of tests performed), or should I only correct at the level of ANOVA?

Thanks a lot in advance,


  • $\begingroup$ @user3614 - the "jerk" response is to do away with p-values, and use a Bayesian answer. Bayes theorem in hypothesis testing automatically does the appropriate corrections you require. Multivariate uniform prior on the effects is the way to go, simulate/integrate (whichever is easier) the effects, then calculate the proportion which fall into the rejection region. Use this as your p-value $\endgroup$ Mar 8, 2011 at 8:00
  • $\begingroup$ @probabilityislogic, best not to confuse others (or yourself) with proselytizing. $\endgroup$
    – cardinal
    Mar 8, 2011 at 17:59
  • 1
    $\begingroup$ @user3614, how many genes? How many subjects? Bonferroni is usually extremely conservative in this setting. $\endgroup$
    – cardinal
    Mar 8, 2011 at 18:02
  • $\begingroup$ @cardinal - touchee. Although from the question as posed, we aren't sure what question @user3614 is trying to answer. p-values give the answer to "how likely is the data, given the effects"? - the Bayesian answer I described gives the answer to "how likely is the true value of the effect sizes, given the data I have?". I would have a guess that the second answer is more appropriate, requiring a Bayesian approach. $\endgroup$ Mar 9, 2011 at 3:23

1 Answer 1


This sound like a hierarchical hypothesis structure, in which you might want to:

  1. Find genes expressed in any treatment
  2. Find the treatment that caused the expression of a given gene.

You will typically go about by answering (1) and then (2). For answering (1), you will want to aggregate over treatments at each gene. "How to aggregate?" depends on the error measure you wish to control. FDR control seems more appropriate in a preliminary study than FWE control. For FDR control, you can use a Simes aggregate p-value (see reference) or be conservative and use the maximal p-value over treatments. Once you have a single p-value for each gene, look for expression using the Benjamini-Hochberg procedure.

Once you have your subset of expressed genes, you can address (2) and look for the treatment that activated them. Assuming you are comfortable with FDR control, you can use a B-H procedure again, but remember to penalize for the fact you have selected those genes! This can be done by using a stricter error level within each gene; Just divide your desired error level by the number of selected genes.

Here is a reference for details.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.