In my biomarker analysis, there are 100 biomarkers. Each biomarker is measured at 4 different time points. I'm looking for any biomarker that is associated with outcome variable at any timepoints. Here the outcome variable is 'Yes' and 'No'. I'm also trying to find any biomarker is expressed differentially at any timepoint at the two outcome levels.

I'm doing the following analysis:

  1. Run 100X4 univariate logistic regressions for all biomarkers for all time points

  2. Run 100x4 t test for all biomarkers for all time points

My question is:

In both cases, should I control family-wise error rate? should I adjust p values?

I really appreciate it if anyone can clarify me on this. Thanks a lot in advance.


1 Answer 1


TL;DR: You may want to consider controlling False Discovery Rate as opposed to Family-wise Error Rate given the number of tests you are doing.

Controlling for a low family-wide error rate for so many analyses could lead to unacceptably low estimation precision for each variable. Biostatisticans deal with this all the time in gene panel studies, so they developed an alternative concept called False Discovery Rate (FDR) as an alternative way to think about the error rates.

From the link above:

The FDR is the rate that features called significant are truly null. An FDR of 5% means that, among all features called significant, 5% of these are truly null.

While this may sound like a p-value, note that it's a post-data assessment -- continued from link above:

Just as we set alpha as a threshold for the p-value to control the FPR [False Positive Rate], we can also set a threshold for the q-value, which is the FDR analog of the p-value. A p-value threshold (alpha) of 0.05 yields a FPR of 5% among all truly null features. A q-value threshold of 0.05 yields a FDR of 5% among all features called significant. The q-value is the expected proportion of false positives among all features as or more extreme than the observed one.

The article I linked to also shows how FDR is connected to the Family-wide Error Rate (FWER) that you have been considering:

The FDR has some useful properties. If all null hypotheses are true (there are no truly alternative results) the FDR=FWER. When there is some number of truly alternative hypotheses, controlling for the FWER automatically also controls the FDR.

  • $\begingroup$ Thanks a lot for the info. Could you elaborate more on my specific case? Specifically how I should adjust my p-value using BH method? @Bey $\endgroup$
    – zesla
    Commented Jan 11, 2019 at 18:37
  • $\begingroup$ @zesla -- you wouldn't -- you'd set a FDR level instead and use q-values. The info in the link is for biostatisticians and applied directly to your type of case. $\endgroup$
    – user233429
    Commented Jan 11, 2019 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.