In the context of high-throughput biological assays (e.g. genomics), where we are applying a hypothesis test (e.g. differential expression between groups of samples) to each analyte in a dataset with many analytes (say, 1000) and some samples (say, 15), we obtain a p-value per analyte (1000 p-values). To correct for the multiple hypothesis tests, we calculate a false discovery rate per analyte by applying the Benjamini-Hochberg procedure to the p-values. As there are many analytes, the FDR is approximately equivalent to the q-value [Storey & Tibshirani, 2003].
We are working with a simulated dataset where we apply a proposed test to each analyte to obtain the p-values, adjust the p-values with FDRs, and threshold the FDRs with a given FDR threshold (say 10%). we can compare the significant analytes to those that we know are actually differential (since we simulated the data) and calculate the empirical FDR as the (number of false discoveries) / (number of analytes called significant). When we simulate multiple such datasets, we obtain multiple independent empirical FDRs.
We would like to summarize these empirical FDRs to assess how well the FDR is being controlling when using my proposed test. One possible summary would be (number of false discoveries over all simulations)/(number of analytes called significant over all simulations), but I don't think this is the quantity we're controlling for, because we're controlling the per simulation FDR. Another line of reasoning is that the FDR or q-value represents P(analyte i is truly null | analyte i is significant) & is an analogue to the p-value [Storey & Tibshirani, 2003]. For independent p-values, an intuitive option is to take the average (mean) of the p-values, but we know independent p-values should not be averaged, because the average p-value will be normally distributed by the Central Limit Theorem, instead of distributed uniformly on [0, 1] under the null, as p-values should be. (However, I can't exclude averaging FDRs, because I don't know their null distribution.) Instead of averaging, Stouffer's method transforms p-values into z-scores, averages the z-scores, and then converts this to a consensus p-value. Applying Stouffer's method to our empirical FDRs seems appealing, but we're not sure it's correct.
Could someone provide feedback on our reasoning or suggest a method for summarizing FDRs/q-values?
References
Storey JD, Tibshirani R. 2003. Statistical significance for genomewide studies. Proc Natl Acad Sci USA. 100(16):9440–45
PS
There are other similar-sounding questions on here, but the answers don't address this question: Combining False Discovery Rates (FDR)? and How to summarize multiple adjusted pvalue into a single measure?.
