I struggling with Benjamini-Hochberg correction due to equally ranked p-values and I would gladly appreciate some help.

My problem is as follow: I've got 11 comparisons to do and the threshold is set at 0.05. My ranked p-values are: p1=0.03125, p2=0.0312, p3=0.0312; p4=0.0312; p5=0.0312; p6=0.0312, p7=0.0312, p8=0.3984, p9=0.5182, p10=0.7912, p11=1.

And the new threshold with Benjamini-correction are : (1) 0.0045, (2) 0.0091; (3) 0.01364; (4) 0.01818; (5) 0.02273; (6) 0.02727; (7) 0.0318; (8) 0.03636; (9) 0.0409; (10) 0.0455; (11) 0.05.

Now if I compare the p-value to the threshold, the first 6 p-values are larger than the p-value so the null hypothesis cannot be rejected. However the seventh p-value is smaller than the corresponding threshold (p7=0.0312<0.0318)… In this case, should the null hypothesis corresponding to the seven first p-value be rejected or not ?


In your example, you would reject the first seven. The BH procedure is a step-up procedure: after ordering your pvalues as $p_1,\dots,p_m$, you should take the last $k$ such that $p_k \leq \frac{k}{m}\alpha$ regardless of whether you have some $p_{k'}$'s with $k'<k$ that do not satisfy this.

Maybe someone else can do a better job of providing intuition, but here's how I think of it. The TL;DR is that what you observed actually happens quite often, and this is perfectly all right: we order the $p_k$'s to get the inverse CDF so that we can approximate the FDR, there is nothing inherently important about the ordering necessarily except that it is computationally easy... we really just need to calculate $\hat{F}_m(u)$, which approximates the (unknown) distribution of the pvalues, and the method you take by ordering does this quickly and easily. To expand on this:

BH works in the following way: suppose you have $\pi$ tests that do reject the null, and thus $1-\pi$ tests that are truly null. Under the null, a pvalue $p$ is uniformly distributed, and so F_{null}(u) = u. The distribution of pvalues that are under the false null have some unknown distribution $F_{notnull}(u)$. Then the distribution of pvalues $p$ is $$F(u) = (1-\pi)F_{null}(u) + \pi F_{nonnull}(u) = (1-\pi)u + \pi F_{nonnull}(u)$$

You can show that for any level $u$ (let me know if you want a proof of this, but it can probably be found online),

$$FDR(u) \approx \frac{(1-\pi)u}{F(u)}$$

and the whole point of FDR is that we want to find $u$ so that $FDR(u) = \alpha$ (which is precisely setting the FDR to level $\alpha$.

We don't know $F(u)$, so we approximate it by ordering the $p_k$'s and have

$$\hat{F}_m(u) = \frac{1}{m} \sum_i 1\{P_i\leq u\}$$

and when we ignore ties (it's easy to adjust for this so it makes no difference, pvalues are continuous, so youll always have some difference between them in reality, and your example still holds if you add some small perturbation and re-order), we have $\hat{F}_m(P_i) = \frac{i}{m}$.

We also don't know $\pi$, so we conservatively set it to $\pi = 0$. Then we have that we want the $p_i$ so that \begin{align*} \alpha = FDR(p_i) & \approx \frac{p_i}{\hat{F}_m(u)} \\ & = \frac{p_im}{i} \\ \end{align*}

and so we want $p_i$ so that $\alpha\frac{i}{m} = p_i$ (or realistically, we want to get as close as possible to this, so we take the max $i$ so that $\alpha\frac{i}{m} \geq p_i$ and reject all the ones below this $p_i$, and so we really don't care about anything before this value.

  • $\begingroup$ Sorry i don’t understand the downvote. If you read my post I make the point that the ordering is unimportant. Also, unless you define step up and step down differently, it certainly is a step up process.. “ the BH procedure is a so-called step-up procedure, that is, if the empirical inverse distribution function crosses the rejection line more than once, all p-values before the last intersection are rejected.” What are you trying to say? (Edit: this was in response to a now removed comment) $\endgroup$ – doubled Jan 22 '20 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.