Correction for Multiple comparisons using Mann-Whitney U test

Question

I am using Mann-Whitney U tests to compare groups of high and low relationship quality on five psychological outcomes. My data is non-normally distributed which is why I am using Mann-Whitney U tests as opposed to independent samples t-tests. From my limited understanding, my design has dependence so is not appropriate for use with Benjamini-Hochberg corrections, is that the case? If so, what alternatives do I have for corrections? I am not keen to use Bonferroni corrections because it is so conservative and the increased risk of type II errors. Thanks

Answer 1

You might be looking for a different answer here, but my opinion is that you should be using Benjamini-Hochberg (potentially using the $q$-value Storey et. al 2002 modification). I'm going to try to show you why.

---

Benjamini-Hochberg (aka. false discovery rate control (FDR)) controls robustly under certain kinds of dependency, e.g. positive regression dependency. What is regression dependency?

From Some Concepts of Dependence Section 5. Regression Dependence, it is shown for two sets $X$ and $Y$ that regression dependence can be written as

$$ P(Y \leq y| X \leq x) \geq P(Y \leq y) $$

meaning that the knowledge of $X$ being small increases the probability of $Y$ being small. This is intuitive; if you eat more food generally you will gain more weight. This is extended slightly to

$$ P(Y \leq y | X = x) $$ is non decreasing in $x$. The Benjamini-Hochberg correction is dependant upon two sets, the set of true null statistics ($I_0$, set of tests which really are not true) and the joint set of test statistics $D$, which is the joint set of true nulls and real associations. In THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY we can look and see that PRDS (positive regression dependence) is then defined as

Property PRDS For any increasing set $D$, and for each $i \in I_0$, $P(X \in D | X_i = x)$ is non-decreasing in $x$.

This intuitively means that for any increasing (ordered) joint statistic distribution, the probability that a test is part of the set of joint (i.e. null plus real, iff there is one real association) is non decreasing as you look at higher $P$-values.

It is also noted that

Therefore, whenever the joint distribution of the test statistics is $PRDS$ on some $I_0$ so is the joint distribution of the corresponding $p$-values, be they right-tailed or left-tailed

If we look at the Mann-Whitney $U$ specifically, we find that for large samples, $U$ is $\sim$ normally distributed. In the case of simple dependencies, your $U$ value will be inflated if it is correlated with a true positive association, so the set of test statistics will be larger. It's intuitive to see that as you as have larger $U$ statistics, you have a greater probability of having a real association.

---

That was kind of long and not really what you were looking for probably, but let's regroup. It's my argument that even though you have a dependency structure, it's probably okay to be using FDR (and unless we have your actual data, which you probably shouldn't give us, then it's best to assume). This is because your data probably follows a condition known as PRDS which is required by FDR correction. You can implement FDR in almost any software package.

If I were you, instead of a list of "significant" or "non-significant" associations, a list of adjusted $q$-values, as detailed by John D. Storey in his paper A direct approach to false disovery rates. In a nut-shell, a $q$-value is the false discovery rate that would be required for your $P$-value to be on the cusp of significance. Perhaps you could present this in tandem with your Bonferroni adjusted $P$-values, because for some reason they make people comfortable.

This is just my opinion, and I would love to hear rebuttals to any part of this. All the best, and thanks for a good question.