In multiple comparisons, suppose we have calculated $q$-values. Is it a $q$-value for each test, or a $q$-value for all the multiple comparisons?

Given a false discovery rate (FDR) threshold on FDR, are $q$-values used to determine whether to accept or reject each null? For an individual test, is it rejected if and only if its $q$-value is no less than the FDR threshold? Why?



It might help to be a little more specific about which FDR you mean, as there are several. One typical False Discovery Rate procedure, due to Benjamini and Hochberg, helps you find an FDR threshold without directly computing $q$ values. it works like this:

  1. Choose an overall $\alpha$ value for the false discovery/false positive rate.
  2. Perform all $m$ individual hypothesis tests ($H_1 \ldots H_m)$, whatever they may be, to produce a list of $p$-values.
  3. Sort the $p$-values, so that $P_{(1)}$ is the smallest and $P_{(m)}$ is the largest. Rearrange the $H$s so the indices match.
  4. Step up from $k=0$ to $k=m$ through your data to find the largest $k$ such that $P_{(k)} \le \frac{k}{m \cdot c(m)}\alpha$. The $c(m)$ is a correction factor, due to Benjamini and Yekutieli, that adjusts for various kinds of dependence/correlation between the tests.
  5. The null hypotheses $H_1 \ldots H_k$ (if any) can be rejected; you fail to reject the remaining null hypothesis (if any).

Note that you don't actually get a $q$ value anywhere in this procedure; it just identifies hypotheses, based on their $p$-values, that you can reject. If you had slightly different data, you might have rejected a null based on a slightly larger $p$ value too. However, you can compute an mean $\alpha$ for your all $m$ tests (for positive dependence, where $c(m)=1$, it's $\bar{\alpha}=\frac{m+1}{2m} \alpha$). Some people also abuse notation slightly and write something like $q<0.05$ (FDR corrected), to distinguish it from uncorrected tests.

However, there are other procedures for controlling the FDR. Storey's approach calculates $q$-values directly. These are intended to be like pFDR adjusted $p$-values, so you would reject null hypotheses associated with $q$-values below some threshold. In that case, you could easily write $q<0.05$.

| cite | improve this answer | |
  • $\begingroup$ Thanks! (1) "be willing to (?) an FDR-corrected null", reject or accept? (2) "feel free to plug in a different 1−α, as necessary", why is it 1−α, not α? $\endgroup$ – Tim Jul 9 '13 at 18:06
  • $\begingroup$ Yes on both accounts. As to why, not enough coffee, I suppose :-) $\endgroup$ – Matt Krause Jul 9 '13 at 18:44
  • $\begingroup$ For (1), the q-value of an individual hypothesis test is the minimum pFDR or minimum FDR at which the test may be called significant. So isn't it that the bigger the q-value is, the more likely the null is rejected? $\endgroup$ – Tim Jul 9 '13 at 18:49
  • $\begingroup$ I am still confused. is a test to reject its null on a sample, if and only if its q-value is no more than the FDR threshold? How do you explain if and explain only if? $\endgroup$ – Tim Jul 9 '13 at 20:46
  • $\begingroup$ Thanks! I was wondering what "FDR corrected" mean and what "uncorrected" mean? What is corrected? $\endgroup$ – Tim Jul 10 '13 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.