# On the applicability of Benjamini-Hochberg

If I've understood correctly, the Benjamini-Hochberg (BH) correction is used to correct for the rate of false discoveries (FDR) when testing a collection of $m$ random variables, $\{X_1, \ldots,X_m\}$ against $m$ null hypotheses $\{H_0^1, \ldots, H_0^m\}$, of which $k\leq m$ can be true.

Now consider a situation where you have a set of random variables that are sorted. For example, $\{Y_1,\ldots,Y_m\}$ with $Y_m\leq\ldots \leq Y_1$ and each $Y_i\sim F_i$ (i.e., distributed according to $F_i$). An example of such a scenario would be the eigenvalues of a random matrix. Suppose now, that given a single null hypothesis $H_0$, and test statistic $T_i$, the $Y_i$'s are tested in descending order — i.e., $Y_1$, then $Y_2$, and so on, until $H_0$ is false.

Is the BH correction also applicable here or is it a fundamentally different scenario? Or does the question of controlling FDR not arise at all?

• (+1), this is interesting, but note that eigenvalues are no more ordered than test statistics are. The real problem is that the eigenvalues are typically not independent before they are sorted. – NRH Aug 20 '12 at 7:55
• @NRH You're right, and often (not always) it is the case that the test statistic $T_i$ used to test $Y_i$ is actually a function of $\{Y_i,\ldots,Y_m\}$ (i.e., the variable being tested and and all smaller ones). I don't know how this would complicate things, but this little bit can be ignored for now if it is confusing. – user13443 Aug 20 '12 at 9:05

• Some questions that arise are — What does the FDR mean in this case? The procedure, if I recall, involves sorting the hypotheses by their $p$ values and then some Boneferroni type of correction is applied to control the FDR at some $q$, based on which the first $k$ hypotheses (sorted by $p$ value) are rejected. Here, one can't move to the next step if the current one isn't rejected. Secondly, a sorting by $p$ value wouldn't make sense. What are we sorting? There is only one hypothesis, but multiple tests with changing datasets each time (a sample is reduced at each step) – user13443 Aug 20 '12 at 19:48
• @user13443 My understanding is that you are test that each Y$_i$ there is a test statistic T$_i$. So you are conducting m tests of a null hypothesis that happen to be dependent because the Ys are ordered. – Michael Chernick Aug 20 '12 at 19:52
• Ok, that's fair. But I don't know how many tests I will end up doing until I fail to reject one (stopping criterion). The remaining $m-k$ tests are not performed at all. So really, it is like doing $k$ tests with $k-1$ of them being rejected and 1 failed to reject, but $k$ being unknown until the iteration stops... – user13443 Aug 20 '12 at 20:21
• That's an interesting way to look at it... not sure it makes sense conceptually (i.e. to the specific problem). Perhaps fail to reject all the remaining $Y_i$s would be more appropriate. Are you aware of any literature that analyzed a similar scenario and discusses FDR/FWER? – user13443 Aug 20 '12 at 21:10