17
$\begingroup$

I have read that controlling FDR is less stringent than controlling FWER, such as in Wikipedia:

FDR controlling procedures exert a less stringent control over false discovery compared to familywise error rate (FWER) procedures (such as the Bonferroni correction). This increases power at the cost of increasing the rate of type I errors, i.e., rejecting the null hypothesis of no effect when it should be accepted.

But I was wondering how it is shown to be true mathematically?

Is there some relation between FDR and FWER?

$\endgroup$
  • 4
    $\begingroup$ Did you read the original paper? It is most everything one could hope for in a statistics paper: A single fundamental idea, a clear and concise story to tell, a useful example, and (short!) accurate proofs. $\endgroup$ – cardinal Mar 27 '15 at 11:15
10
+50
$\begingroup$

Indeed, @cardinal is quite right that the paper is as clear as it gets. So, for what it's worth, in case you do not have access to the paper, here's a slightly elaborated version of how Benjamini–Hochberg argue:

The FDR $Q_e$ is the expected value of the proportion of false rejections $v$ to all rejections $r$. Now, $r$ is, obviously, the sum of false and correct rejections; call the latter $s$.

In summary, (using capital letters for random variables and lowercase letters for realized values),

$$Q_e=E\left(\frac{V}{R}\right)=E\left(\frac{V}{V+S}\right)=:E\left(Q\right).$$

One takes $Q=0$ if $R=0$.

Now, there are two possibilities: either all $m$ nulls are true or just $m_0<m$ of them are true. In the first case, there cannot be correct rejections, so $r=v$. Thus, if there are any rejections ($r\geq 1$), $q=1$, otherwise $q=0$. Hence,

$$\newcommand{\FDR}{\mathrm{FDR}}\newcommand{\FWER}{\mathrm{FWER}}\FDR=E(Q)=1\cdot P(Q=1)+0\cdot P(Q=0)=P(Q=1)=P(V \geq 1)=\FWER$$

So, $\FDR=\FWER$ in this case, such that any procedure that controls the $\FDR$ trivially also controls the $\FWER$ and vice versa.

In the second case in which $m_0<m$, if $v>0$ (so if there is at least one false rejection), we obviously have (this being a fraction with also $v$ in the denominator) that $v/r\leq 1$. This implies that the indicator function that takes the value 1 if there is at least one false rejection, $\mathbf 1_{V\geq 1}$ will never be less than $Q$, $\mathbf 1_{V\geq 1}\geq Q$. Now, take expectation on either side of the inequality, which by monotonicity of $E$ leaves the inequality intact,

$$E(\mathbf 1_{V\geq 1})\geq E(Q)=\FDR$$

The expected value of an indicator function being the probability of the event in the indicator, we have $E(\mathbf 1_{V\geq 1})=P(V\geq 1)$, which again is the $\FWER$.

Thus, when we have a procedure that controls the $\FWER$ in the sense that $\FWER\leq \alpha$, we must have that $\FDR\leq\alpha$.

Conversely, having $\FDR$ control at some $\alpha$ may come with a substantially larger $\FWER$. Intuitively, accepting a nonzero expected fraction of false rejections ($\FDR$) out of a potentially large total of hypotheses tested may imply a very high probability of at least one false rejection ($\FWER$).

So, a procedure has to be less strict when only $\FDR$ control is desired, which is also good for power. This is the same idea as in any basic hypothesis test: when you test at the 5% level you reject more frequently (both correct and false nulls) than when testing at the 1% level simply because you have a smaller critical value.

$\endgroup$
  • 2
    $\begingroup$ (+1) Good exposition. Obviously, in the first case we can also say FWER control implies FDR control (which is the matter in question). Also, it may be worth pointing out that this property comes with no distributional assumptions (e.g., independence) on the test statistics, unlike the procedure given in the original paper for control of the FDR. $\endgroup$ – cardinal Mar 28 '15 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.