It's common practice to adjust for multiple testing when analyzing high-throughput gene expression data, eg. RNA-seq and microarrays. Despite this, researchers routinely select a subset of genes for further investigation, let's say the top 90 genes from high-throughput, and run a qPCR experiment on those, for example.

My questions is: Should P-values obtained from omnibus tests (e.g. ANOVA and its non-parametric counterparts) be adjusted for multiple testing (given that there are 90 tests/genes). If so, which procedure would deal with the possible dependence between gene expression in this case?

Also, there's a second more complicated scenario, which is about multiple testing correction and multiple comparison adjustment. Sometimes, besides the 90 genes, the researcher also wants to compare multiple groups pairwise, which would fall into the multiple comparison scenario within the multiple-testing (several genes, or variables) case. How to deal with this hierarchichal structure?

Biomedical literature is flooded with multiple T-tests performed across multiple genes/vars. not being adjusted and also the examples described above.

Any input is appreaciated.




1 Answer 1


Yes, correction for multiple hypothesis testing is important, but you have to consider it in the context of how these types of biomedical investigations can (and should) proceed. One important issue is that, in my experience, most experimental scientists think like Bayesians even though they typically are required to perform frequentist tests for statistical significance.

Lets start with RNA-seq or microarray data to determine expression of messenger RNA (mRNA), in which you might be examining up to 20,000 genes in a set of different experimental settings. These usually represent initial steps to generate hypotheses about biological processes that then will be studied in more experimental detail.

As these are used for discovery, the false discovery rate (FDR) is the best way to deal with multiple comparisons. You accept that some of your "significant" findings will be erroneous, but take that as an acceptable tradeoff for having power to find more potentially important leads. In contrast, if you attempt to control the family-wise error rate, you are penalized if any of your "significant" results might be erroneous, which makes it much harder to find true positive results.

So say you have a list of 90 genes that passed a 10% FDR cutoff in RNA-seq readings in Treatment T versus control conditions C. You need to validate these 90 results in 2 ways. First, you want to make sure that there wasn't something artifactual in the RNA-seq analysis, so you use a different analysis method for mRNA, like quantitative RT-PCR (qRT-PCR). Second, you have to deal with the 9 or so genes that might be false positives.

This is where experimental scientists think like Bayesians: they already have evidence that these 90 genes are significantly affected by Treatment T, and they are now looking for further information to increase or decrease their belief in these relationships. Standard frequentist tests of significance on the 90 qRT-PCR results and corrections for multiple testing, however, are typically based on the null hypothesis that Treatment T has no effect for any of them--even though we know the null hypothesis is very unlikely given the RNA-seq results already in hand.

Furthermore, as you imply, some of these 90 genes are likely to be co-regulated in biological control pathways. Results of RNA-seq or qRT-PCR among co-regulated genes are not expected to be independent; standard multiple-hypothesis correction, which assumes independence among the hypotheses, would thus be too stringent.

Finally, statistical tests of RNA-seq and validating qRT-PCR data are often not the end result of this type of study. The results of statistical hypothesis testing on those data are best used for generating scientific hypotheses as an initial part of a broader study. The study goes on to use tools of cellular and molecular biology to investigate the underlying cellular and biochemical pathways in ways that challenge those scientific hypotheses. In this type of research, if well designed, tests of a scientific hypothesis should be robust to false-positive results for any single statistical test.

So it might be that two "wrongs" potentially make a "right" in this type of research. Yes, it is wrong to ignore corrections for simultaneously testing multiple hypotheses, as for your 90 qRT-PCR results. But it also can be wrong to base statistical tests on null hypotheses that are already known to be unlikely, or to use a test that assumes independence among a set of hypotheses that almost certainly are not independent. That's why it's more important to focus on the evidence for the scientific hypotheses about the cellular and molecular mechanisms at work than to worry too much about some details of multiple hypothesis testing.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.