How is power for t test and Mann Whitney comparable when null hypotheses differ

Question

In a previous question, I asked about comparing the power of a t test to a Mann Whitney test under different situations. One of the answers pointed out that the worst that the Mann-Whitney can ever perform relative to the t-test is that it would require 1/0.864x as much data to give the same power as the t test so long as the data sets being compared were from the same distribution.

I guess I am confused about how power can be compared between these two tests given that they test different null hypotheses. For example, if I estimate the power for a t-test at 0.9 using effect size equal to a 20% difference between means, then that makes sense for a t test. But a Mann Whitney test does not test for differences between means. If the distributions were the same it would test for differences between medians. Am I right in thinking that the Mann Whitney test would require 1/0.864x as much data to detect a 20% difference in medians with a power of 0.9?

Answer 1

1) It's important to distinguish between an estimator and the population quantity it is being used to estimate.

For example, for a symmetric distribution whose mean exists, I could use any number of sample estimators of the population mean - such as the sample median, for example. [And, for example, if the distribution is Laplace, the sample median is actually a substantially better estimate of the population mean than the sample mean is.]

2) If you assume the same distribution-shape (i.e. identical apart from a possible location shift) for the two populations in your test, then both the Mann-Whitney and the t-test test for that location shift, which is the same shift whether you think of it as a shift in means, medians, 95th percentiles or whatever.

In that situation, the Mann-Whitney is as much a test of difference in populations means as the t-test is, it just uses a different estimator for the quantity. Indeed, it's quite possible to produce both an estimate of (and interval for) that shift under either test.

The Mann-Whitney estimate of the shift is the median of pairwise cross-sample differences (see the second paragraph here) -- but similar to the example in (1), that's a perfectly valid way to estimate the shift in population means.

For more discussion of some of these issues, and how a Mann-Whitney and a t-test test the same null and alternative hypothesis once you assume that the alternative is location-shift, see this answer