10
$\begingroup$

So a Mann Whitney U test is supposedly about 95% as powerful as a t-test when t-test assumptions of normality and homogeneous variance are satisfied. I also know that a Mann Whitney U test is more powerful than a t-test when these assumptions are not satisfied. My question is, is a Mann Whitney test on data where assumptions aren't satisfied as or almost powerful as a t-test on data where assumptions are satisfied?

I'm asking because I often see people doing power calculations based on the assumption that they will perform a t test. After they collect the data, they explore data and decide to use a Mann Whitney test instead and don't really revisit how changing the test affected power.

Thanks!

Improve this question
$\endgroup$
1
  • $\begingroup$ "I also know that a Mann Whitney U test is more powerful than a t-test when these assumptions are not satisfied". That's too strong a statement. Let's say the data were uniformly distributed (for example). You say you know that in those circumstances a U test is more powerful than the t, but it's not the case. $\endgroup$
    – Glen_b
    Sep 28, 2013 at 0:03

2 Answers 2

Reset to default
8
$\begingroup$

1) The Mann-Whitney test is not guaranteed to be more powerful than a t-test when the assumptions of the t-test are not satisfied, although for the sorts of violations we tend to see in the real world, it is. Consider a standard Normal distribution truncated at +/- 100 and a difference between means of two groups of 0.01; this isn't Normal, but both tests will perform as if it were, since the difference between the two distributions is so small.

2) The t-test is the uniformly most powerful test for difference between means of two Normal variates blah blah blah, so it isn't going to be beaten by the Mann-Whitney on that kind of data no matter what. However, the worst that the Mann-Whitney can ever perform relative to the t-test is about 0.864 in terms of asymptotic relative efficiency, i.e., it would require 1/0.864x as much data to give the same power (asymptotically.) (Hollander and Wolfe, Nonparametric Statistical Methods.) There isn't any bound going the other way. Reproducing some numbers from Hollander and Wolfe, for different distributions we get an A.R.E. of the M-W to the t-test of:

  1. Normal: 0.955
  2. Uniform: 1.0 <- also a counterexample to the M-W being better than the t for non-normal dist'ns
  3. Logistic: 1.097
  4. Double Exponential: 1.5
  5. Exponential: 3.0
  6. Cauchy (well that's easy): $\infty$

The point of course being that you can't shoot yourself in the foot by using the Mann-Whitney test instead of the t-test, but the converse is not true.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Why is Cauchy easy? And why is ARE 0? For finite N, the relative efficiency can't be $\inf$, because the power of the t-test isn't 0. But with infinite N, the variance of the distribution is undefined. That Cauchy sure is perverse! $\endgroup$
    – Peter Flom
    Sep 28, 2013 at 0:36
  • $\begingroup$ @PeterFlom Interesting indeed! The difference between the limiting value and the value at the limit rears its head; the Pitman ARE is the former, not the latter. $\endgroup$
    – jbowman
    Sep 28, 2013 at 0:55
  • 2
    $\begingroup$ @PeterFlom ARE relates to the ratio of the second derivatives ("curvature") of the power curves at the null, as sample size goes to infinity. It's possible for a power curve to have zero second derivative there. In practice, small to moderate-size samples, the two sample t does kind of okay at the Cauchy if you don't mind your significance levels being a lot lower than the nominal values. $\endgroup$
    – Glen_b
    Sep 28, 2013 at 1:24
  • $\begingroup$ So in other words, let's say I was lazy and didn't want to check my assumptions of normality etc. and just decided to go ahead and use a MW test instead of a t-test. I could use the MW test and say that, at worst I would need 1/0.864x as much data to achieve that same level of power as a t test where all assumptions were met. Does that make sense? $\endgroup$
    – Jimj
    Sep 28, 2013 at 5:45
  • 1
    $\begingroup$ @Jimj no, that's not what it means. You could use the MW test and say that (in large samples) at worst you would need 1/0.864 times as much data to achieve that same level of power as a t test on data sets from the same distribution (that 0.864 doesn't happen when all the assumptions of the t are met... when they are, the ARE is 0.955) $\endgroup$
    – Glen_b
    Sep 28, 2013 at 11:45
10
$\begingroup$

is a Mann Whitney test on data where assumptions aren't satisfied as or almost powerful as a t-test on data where assumptions are satisfied?

A phrase like 'as powerful' doesn't really work as a general statement.

Power isn't especially comparable across different distributional models. The size of a given effect has different meanings in different parts of the distribution. Imagine you have a distribution that is pretty peaked, but has a heavy tail; by what measure to we say a particular size of deviation is similar to something with a much 'flatter' centre and smaller tail? A small deviation might be about as easy to pick up, but a large deviation might be (relative to the other distributional possibility we're trying to compare power for) harder.

With two possible sets of normal distributions, one pair with a large s.d. and one with a small s.d., it's easy to say 'well, power will just scale with standard deviation; if we define our effect size in terms of number of standard deviations, we can relate the two power curves'.

But now with differently shaped distributions, there's no obvious scale choice. We must make some choices about how to compare them. What choices we make will determine how they "compare".

For example, how do I compare power when the data are Cauchy with power when the data are say a scaled beta(2,2)? What is a comparable effect size? The Cauchy below has more of its distribution between -1 and 1 and less of its distribution between -3 and 3 than the other one. Their interquartile ranges are different, for example. What is our basis for comparison?

Cauchy vs scaled beta

If you can resolve that conundrum, now consider if one of the distributions is skewed left and the other is bimodal, or any of a myriad number of other possibilities.

You can still compute power under any particular set of assumptions, but comparison of one test across different distributional assumptions rather than two tests under a given distributional assumption is conceptually very tricky.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.