# Multiple testing, FDR and power

Suppose that I've a data with a total of 30,000 observations. Each observation has a total of 6 values, 3 per condition. I'm interested in, let's say, testing if the means of these two conditions for each of these observations are different or not. For this, we typically apply a statistical test of choice to each of these 30K observations. In order to control the false positive rate due to multiple testing, we'd control for FDR at say 5% using BH method.

Example of an observation would be:

# dummy data
Condition 1: 40, 55, 48
Condition 2: 129, 77, 181


Also, let's assume I obtained a total of 1000 "significant" results. If I filter the data somehow, say, remove all those observations where the maximum value across all 6 values is < 5 and end up with a filtered set of 20K observations, then I find that the number of significant results is a bit higher, say 1250. If I continue this approach of filtering data with higher and higher thresholds (< 10, < 20 ... preferably using quantile = 0.1, 0.2, 0.3 etc..), I find that the number of significant results at 5% FDR keeps increasing up to a certain point and with too stringent filtering starts to reduce again.

It's fairly obvious that in controlling the FP in a multiple testing setup one compromises for the statistical power. My question is, are there methods that could somehow compute a filtering criteria which maximises the power (to detect after FDR correction)? If not, is it statistically sound approach to try and filter data with more than 1 value and decide on the one that maximises significance?

Applying a filter to the data is clearly a case of selective inference (a.k.a. double dipping, data snooping, etc.) If you are removing true null hypotheses, you will have an anti-conservative FDR. If you are removing true alternative hypotheses, you will have a conservative FDR.

The filtering criterion you are using seems "fair" a-priori in the sense it does not seem to be correlated with the truthfulness of a hypothesis. On the other hand, the fact that you are making more and more discoveries suggests that high response values, might also be correlated with a difference between groups. I thus suspect your filtering step will have increased the FDR of the BH procedure. Try testing it using a simulation.

Also, could it be you are using an analysis that assumes homoscedasticity where the variances actually grow with the means?. If this is the case, try some non-parametric test to compare the groups.

Finally, back to the original question: I guess it should be possible to find the optimal number of hypotheses to test using the BH procedure. You will naturally need to define a generalized notion of power (say, average power over hypotheses). You will need to know the proportion of true null hypotheses and the power of each marginal test.

• JohnRos, thanks for your reply. Could you please elaborate on the line: "On the other hand, the fact that you're making more and more discoveries suggest that..."? – Arun Jul 17 '13 at 8:00
• We assume that the counts follow a NB distribution. And yes, we assume homoscedasticity and with NB, var > mean. And variance is modeled as a quadratic function of mean. So, yes, variance increases with mean. – Arun Jul 17 '13 at 8:06
• One more point: In this paper, in figure 1c, they show the effect of using 2 different filters at different thresholds (x-axis) against the number of NULL hypothesis rejected. And you see that the variance method (black lines), which is a better filter (acc. to this paper) rejects more NULL with increasing filtering. – Arun Jul 17 '13 at 8:14
• "... you're making more and more discoveries...": A known phenomenon with FDR is that you can gain power by adding false null hypotheses, or removing true nulls [1] (referred to as "cheating with FDR"). This is not necessarily your case, just a suspicion of mine. [1] Finner, Helmut, and M. Roters. “On the False Discovery Rate and Expected Type I Errors.” Biometrical Journal 43, no. 8 (2001): 985–1005. doi:10.1002/1521-4036(200112)43:8<985::AID-BIMJ985>3.0.CO;2-4. – JohnRos Jul 17 '13 at 8:17
• By simulation I mean- apply your filter+BH procedure on simulated data to check that the FDR is controlled. – JohnRos Jul 17 '13 at 10:26