# False discovery rate & q-values: how are q-values to be interpreted when rank of p-values is altered?

I am trying to wrap my head around False Discovery Rate, and its associated q-value; I am new to this technique, but it seems quite promising for my needs.

One sticking point I keep coming across and cannot seem to resolve is the following:

As best as I can tell, the ordering of q-values can be different from that of p-values, which seems nonsense to me. Further explaining:

q-value is calculated by taking a sorted ordering of p-values and adjusting those p-values by the rank. E.g., in the original Benjamini & Hochberg paper (Controlling False Discovery Rate: A practical and Powerful Approach to Multiple Testing), a value q* is introduced which is

 (# tests) * p-sub-i / (rank order of p-sub-i)


Story later introduces a correction factor "pi0" to correct for the fact that not all sampled tests are null tests.

Using these calculations (either version), one can then "order" the q-values, and have some sense of which tests were significant, relative to other tests, while still considering the crux of multiple test comparisons.

However, if p-values are quite close to each other, as often occurs, this rank ordering can switch. E.g., if p-values are 0.00505 (a) and 0.00506 (b), for instance, q-values can turn out to be 0.012 (a) and 0.011 (b), respectively. The reason for this is that the change in the q-value due to the increased index, or ranking, of the p-value might be more significant than the change in the p-value itself.

The example shown above is quite small, but nonetheless points to a theoretical implication I don't understand: some tests with lower p-values than other tests may end up with higher q-values than other tests, implying that null hypothesis tests are "arbitrarily" be impacted by their neighbors.

What am I missing here?

With $q$ fixed, we set $P_{(i)}$ to be the the sorted p-values. The method then rejects $H_{(i)}$ for $i = 1,2,\ldots,k$ where $k$ is the largest $i$ for which $P_{(i)} \leq \frac{i}{m}q$.
• To add to the confusion, sometimes the term "$q$-value" is used to mean "$p$-values that have been adjusted by the multiple comparisons procedure". That's the meaning that I read the OP's question in light of. Apr 25, 2014 at 16:16
The reason for the change in order is that the q-value measures a fundamentally different thing than the p-value. q-value is the false detection rate (FDR) at a given level of statistical significance. Let's say your 5th lowest observed p-value was 0.02 and by using some statistical method you estimated that you would get on average 2 false positive detections by using this threshold for significance. This would give an FDR of $$2/5 = 0.4$$. Let's then say that the 8th lowest p-value was 0.045 and the estimated number of false positives was 3. Then $$q = 3/8 = 0.375$$. The order of q-values doesn't match the p-values.