I am using Mann-Whitney U tests to compare groups of high and low relationship quality on five psychological outcomes. My data is non-normally distributed which is why I am using Mann-Whitney U tests as opposed to independent samples t-tests. From my limited understanding, my design has dependence so is not appropriate for use with Benjamini-Hochberg corrections, is that the case? If so, what alternatives do I have for corrections? I am not keen to use Bonferroni corrections because it is so conservative and the increased risk of type II errors. Thanks

  • $\begingroup$ What are the outcomes? Are they 5 different measures of a single underlying construct? Also, you would do best not to dichotomize relationship quality. $\endgroup$ Commented Aug 19, 2015 at 2:33
  • $\begingroup$ They are self-concept, physical self-concept, depressive symptoms, anxiety symptoms, and illness acceptance. I agree with you but for the purposes of my experiment and what my supervisor wanted it is dichotomised $\endgroup$
    – Tom
    Commented Aug 19, 2015 at 2:42
  • $\begingroup$ Benjamini-Hochberg is robust to certain kinds of dependency. See my answer here $\endgroup$
    – Chris C
    Commented Aug 19, 2015 at 2:56
  • $\begingroup$ Thanks @ChrisC but I am a bit out of my depth and not sure if my study is an example of dependency that is acceptable for the Bejamini-Hochberg? $\endgroup$
    – Tom
    Commented Aug 19, 2015 at 3:07
  • $\begingroup$ would a holm correction be applicable? $\endgroup$
    – Tom
    Commented Aug 19, 2015 at 3:35

1 Answer 1


You might be looking for a different answer here, but my opinion is that you should be using Benjamini-Hochberg (potentially using the $q$-value Storey et. al 2002 modification). I'm going to try to show you why.

Benjamini-Hochberg (aka. false discovery rate control (FDR)) controls robustly under certain kinds of dependency, e.g. positive regression dependency. What is regression dependency?

From Some Concepts of Dependence Section 5. Regression Dependence, it is shown for two sets $X$ and $Y$ that regression dependence can be written as

$$ P(Y \leq y| X \leq x) \geq P(Y \leq y) $$

meaning that the knowledge of $X$ being small increases the probability of $Y$ being small. This is intuitive; if you eat more food generally you will gain more weight. This is extended slightly to

$$ P(Y \leq y | X = x) $$ is non decreasing in $x$. The Benjamini-Hochberg correction is dependant upon two sets, the set of true null statistics ($I_0$, set of tests which really are not true) and the joint set of test statistics $D$, which is the joint set of true nulls and real associations. In THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY we can look and see that PRDS (positive regression dependence) is then defined as

Property PRDS For any increasing set $D$, and for each $i \in I_0$, $P(X \in D | X_i = x)$ is non-decreasing in $x$.

This intuitively means that for any increasing (ordered) joint statistic distribution, the probability that a test is part of the set of joint (i.e. null plus real, iff there is one real association) is non decreasing as you look at higher $P$-values.

It is also noted that

Therefore, whenever the joint distribution of the test statistics is $PRDS$ on some $I_0$ so is the joint distribution of the corresponding $p$-values, be they right-tailed or left-tailed

If we look at the Mann-Whitney $U$ specifically, we find that for large samples, $U$ is $\sim$ normally distributed. In the case of simple dependencies, your $U$ value will be inflated if it is correlated with a true positive association, so the set of test statistics will be larger. It's intuitive to see that as you as have larger $U$ statistics, you have a greater probability of having a real association.

That was kind of long and not really what you were looking for probably, but let's regroup. It's my argument that even though you have a dependency structure, it's probably okay to be using FDR (and unless we have your actual data, which you probably shouldn't give us, then it's best to assume). This is because your data probably follows a condition known as PRDS which is required by FDR correction. You can implement FDR in almost any software package.

If I were you, instead of a list of "significant" or "non-significant" associations, a list of adjusted $q$-values, as detailed by John D. Storey in his paper A direct approach to false disovery rates. In a nut-shell, a $q$-value is the false discovery rate that would be required for your $P$-value to be on the cusp of significance. Perhaps you could present this in tandem with your Bonferroni adjusted $P$-values, because for some reason they make people comfortable.

This is just my opinion, and I would love to hear rebuttals to any part of this. All the best, and thanks for a good question.

  • 1
    $\begingroup$ I guess they made an error in Wikipedia: "The Benjamini–Hochberg–Yekutieli procedure controls the false discovery rate under positive dependence assumptions.", en.wikipedia.org/wiki/False_discovery_rate#Benjamini–Hochberg–Yekutieli_procedure . BH is enough for that. I am writing this comment because probably smb else will be confused by wiki and this answer was the next link on google. $\endgroup$ Commented Jan 17, 2018 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.