Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite
1

If I've understood correctly, the Benjamini-Hochberg (BH) correction is used to correct for the rate of false discoveries (FDR) when testing a collection of $m$ random variables, $\{X_1, \ldots,X_m\}$ against $m$ null hypotheses $\{H_0^1, \ldots, H_0^m\}$, of which $k\leq m$ can be true.

Now consider a situation where you have a set of random variables that are sorted. For example, $\{Y_1,\ldots,Y_m\}$ with $Y_m\leq\ldots \leq Y_1$ and each $Y_i\sim F_i$ (i.e., distributed according to $F_i$). An example of such a scenario would be the eigenvalues of a random matrix. Suppose now, that given a single null hypothesis $H_0$, and test statistic $T_i$, the $Y_i$'s are tested in descending order — i.e., $Y_1$, then $Y_2$, and so on, until $H_0$ is false.

Is the BH correction also applicable here or is it a fundamentally different scenario? Or does the question of controlling FDR not arise at all?

share|improve this question
1  
(+1), this is interesting, but note that eigenvalues are no more ordered than test statistics are. The real problem is that the eigenvalues are typically not independent before they are sorted. – NRH Aug 20 '12 at 7:55
@NRH You're right, and often (not always) it is the case that the test statistic $T_i$ used to test $Y_i$ is actually a function of $\{Y_i,\ldots,Y_m\}$ (i.e., the variable being tested and and all smaller ones). I don't know how this would complicate things, but this little bit can be ignored for now if it is confusing. – user13443 Aug 20 '12 at 9:05

2 Answers

up vote 1 down vote

Sorted test statistics are not a problem for B-H as the (re)sorting is an inherent part of the procedure. In the eigenvalue case-- when testing for the rank of the eigenspace-- it is actually the dependence between the eigenvalues that is more problematic. I am not sure if the PRDS condition needed for B-H is satisfied, so you could either consider a more general FDR controlling procedure such as B-H-Y, or have a look at some works on the distribution of eigen values (say, B. Nadler and I.M. Johnstone Detection Performance of Roy's Largest Root Test when the noise covariance matrix is arbitrary, Statistical Signal Processing Conference, Nice, France, 2011.).

share|improve this answer
up vote 0 down vote

Instead of independent tests you have dependent tests. But multiple testing concepts like familywise error rate (FWER) and false discovery rate (FDR) still apply. The complication is in the computation of quantities like FWER as probabilities simultaneously rejecting two of the hypotheses is no longer the product of the individual rejection probabilities.

share|improve this answer
Some questions that arise are — What does the FDR mean in this case? The procedure, if I recall, involves sorting the hypotheses by their $p$ values and then some Boneferroni type of correction is applied to control the FDR at some $q$, based on which the first $k$ hypotheses (sorted by $p$ value) are rejected. Here, one can't move to the next step if the current one isn't rejected. Secondly, a sorting by $p$ value wouldn't make sense. What are we sorting? There is only one hypothesis, but multiple tests with changing datasets each time (a sample is reduced at each step) – user13443 Aug 20 '12 at 19:48
@user13443 My understanding is that you are test that each Y$_i$ there is a test statistic T$_i$. So you are conducting m tests of a null hypothesis that happen to be dependent because the Ys are ordered. – Michael Chernick Aug 20 '12 at 19:52
Ok, that's fair. But I don't know how many tests I will end up doing until I fail to reject one (stopping criterion). The remaining $m-k$ tests are not performed at all. So really, it is like doing $k$ tests with $k-1$ of them being rejected and 1 failed to reject, but $k$ being unknown until the iteration stops... – user13443 Aug 20 '12 at 20:21
@user13443 Actually i think you can view it that you perform all the tests. It just happens that once you reject the null hypothesis you continue to rejec6t for all the rremaining Yis. – Michael Chernick Aug 20 '12 at 20:27
That's an interesting way to look at it... not sure it makes sense conceptually (i.e. to the specific problem). Perhaps fail to reject all the remaining $Y_i$s would be more appropriate. Are you aware of any literature that analyzed a similar scenario and discusses FDR/FWER? – user13443 Aug 20 '12 at 21:10
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.