6
$\begingroup$

I am trying to wrap my head around False Discovery Rate, and its associated q-value; I am new to this technique, but it seems quite promising for my needs.

One sticking point I keep coming across and cannot seem to resolve is the following:

As best as I can tell, the ordering of q-values can be different from that of p-values, which seems nonsense to me. Further explaining:

q-value is calculated by taking a sorted ordering of p-values and adjusting those p-values by the rank. E.g., in the original Benjamini & Hochberg paper (Controlling False Discovery Rate: A practical and Powerful Approach to Multiple Testing), a value q* is introduced which is

 (# tests) * p-sub-i / (rank order of p-sub-i)

Story later introduces a correction factor "pi0" to correct for the fact that not all sampled tests are null tests.

Using these calculations (either version), one can then "order" the q-values, and have some sense of which tests were significant, relative to other tests, while still considering the crux of multiple test comparisons.

However, if p-values are quite close to each other, as often occurs, this rank ordering can switch. E.g., if p-values are 0.00505 (a) and 0.00506 (b), for instance, q-values can turn out to be 0.012 (a) and 0.011 (b), respectively. The reason for this is that the change in the q-value due to the increased index, or ranking, of the p-value might be more significant than the change in the p-value itself.

The example shown above is quite small, but nonetheless points to a theoretical implication I don't understand: some tests with lower p-values than other tests may end up with higher q-values than other tests, implying that null hypothesis tests are "arbitrarily" be impacted by their neighbors.

What am I missing here?

$\endgroup$

2 Answers 2

2
$\begingroup$

The method described in the Benjami-Hochberg paper does not have multiple q-values. What do you mean by "one can then 'order' the q-values?'. One fixes at the onset, a q-value (say 0.05). This means we want to control the FDR at the level q. That is, the expected ratio of incorrectly rejected to rejected hypothesis will be less than q.

With $q$ fixed, we set $P_{(i)}$ to be the the sorted p-values. The method then rejects $H_{(i)}$ for $i = 1,2,\ldots,k$ where $k$ is the largest $i$ for which $P_{(i)} \leq \frac{i}{m}q$.

$\endgroup$
2
  • $\begingroup$ Yes, for the original B-H paper, you are correct, the q-value is specified ahead of time. Looking at a Storey paper (e.g., "The Positive FDR: A Bayesian Interpret. and the q-value"), he makes the argument that you can essentially solve for q(i's) instead of P(i's). <br><br> Regardless, even in your equation above, you'll still see multiple P(i's), which can run into the identical problem I describe above. $\endgroup$ Jul 28, 2011 at 23:48
  • 1
    $\begingroup$ To add to the confusion, sometimes the term "$q$-value" is used to mean "$p$-values that have been adjusted by the multiple comparisons procedure". That's the meaning that I read the OP's question in light of. $\endgroup$
    – Alexis
    Apr 25, 2014 at 16:16
1
$\begingroup$

The reason for the change in order is that the q-value measures a fundamentally different thing than the p-value. q-value is the false detection rate (FDR) at a given level of statistical significance. Let's say your 5th lowest observed p-value was 0.02 and by using some statistical method you estimated that you would get on average 2 false positive detections by using this threshold for significance. This would give an FDR of $2/5 = 0.4$. Let's then say that the 8th lowest p-value was 0.045 and the estimated number of false positives was 3. Then $q = 3/8 = 0.375$. The order of q-values doesn't match the p-values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.