Suppose that I've a data with a total of 30,000 observations. Each observation has a total of 6 values, 3 per condition. I'm interested in, let's say, testing if the means of these two conditions for each of these observations are different or not. For this, we typically apply a statistical test of choice to each of these 30K observations. In order to control the false positive rate due to multiple testing, we'd control for FDR at say 5% using BH method.

Example of an observation would be:

# dummy data
Condition 1: 40, 55, 48
Condition 2: 129, 77, 181

Also, let's assume I obtained a total of 1000 "significant" results. If I filter the data somehow, say, remove all those observations where the maximum value across all 6 values is < 5 and end up with a filtered set of 20K observations, then I find that the number of significant results is a bit higher, say 1250. If I continue this approach of filtering data with higher and higher thresholds (< 10, < 20 ... preferably using quantile = 0.1, 0.2, 0.3 etc..), I find that the number of significant results at 5% FDR keeps increasing up to a certain point and with too stringent filtering starts to reduce again.

It's fairly obvious that in controlling the FP in a multiple testing setup one compromises for the statistical power. My question is, are there methods that could somehow compute a filtering criteria which maximises the power (to detect after FDR correction)? If not, is it statistically sound approach to try and filter data with more than 1 value and decide on the one that maximises significance?


1 Answer 1


Applying a filter to the data is clearly a case of selective inference (a.k.a. double dipping, data snooping, etc.) If you are removing true null hypotheses, you will have an anti-conservative FDR. If you are removing true alternative hypotheses, you will have a conservative FDR.

The filtering criterion you are using seems "fair" a-priori in the sense it does not seem to be correlated with the truthfulness of a hypothesis. On the other hand, the fact that you are making more and more discoveries suggests that high response values, might also be correlated with a difference between groups. I thus suspect your filtering step will have increased the FDR of the BH procedure. Try testing it using a simulation.

Also, could it be you are using an analysis that assumes homoscedasticity where the variances actually grow with the means?. If this is the case, try some non-parametric test to compare the groups.

Finally, back to the original question: I guess it should be possible to find the optimal number of hypotheses to test using the BH procedure. You will naturally need to define a generalized notion of power (say, average power over hypotheses). You will need to know the proportion of true null hypotheses and the power of each marginal test.

  • $\begingroup$ JohnRos, thanks for your reply. Could you please elaborate on the line: "On the other hand, the fact that you're making more and more discoveries suggest that..."? $\endgroup$
    – Arun
    Commented Jul 17, 2013 at 8:00
  • $\begingroup$ We assume that the counts follow a NB distribution. And yes, we assume homoscedasticity and with NB, var > mean. And variance is modeled as a quadratic function of mean. So, yes, variance increases with mean. $\endgroup$
    – Arun
    Commented Jul 17, 2013 at 8:06
  • $\begingroup$ One more point: In this paper, in figure 1c, they show the effect of using 2 different filters at different thresholds (x-axis) against the number of NULL hypothesis rejected. And you see that the variance method (black lines), which is a better filter (acc. to this paper) rejects more NULL with increasing filtering. $\endgroup$
    – Arun
    Commented Jul 17, 2013 at 8:14
  • 2
    $\begingroup$ "... you're making more and more discoveries...": A known phenomenon with FDR is that you can gain power by adding false null hypotheses, or removing true nulls [1] (referred to as "cheating with FDR"). This is not necessarily your case, just a suspicion of mine. [1] Finner, Helmut, and M. Roters. “On the False Discovery Rate and Expected Type I Errors.” Biometrical Journal 43, no. 8 (2001): 985–1005. doi:10.1002/1521-4036(200112)43:8<985::AID-BIMJ985>3.0.CO;2-4. $\endgroup$
    – JohnRos
    Commented Jul 17, 2013 at 8:17
  • 1
    $\begingroup$ By simulation I mean- apply your filter+BH procedure on simulated data to check that the FDR is controlled. $\endgroup$
    – JohnRos
    Commented Jul 17, 2013 at 10:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.