7
$\begingroup$

What does it mean when people say that a t-test performed on ranked data is equivalent to a Mann-Whitney U-test? Does that mean they just test the same hypothesis/are useful in the same situations or are they are supposed to give the exact same p-values? The reason I ask is I tried both in R and compared two groups with very small sample sizes (3 and 4). I got completely different answers: one significant and one not.

The two groups are A=(1,2,3) and B=(4,5,6,7).

t-test: p = 0.01

Mann-Whitney U-test: p = 0.06

$\endgroup$
  • $\begingroup$ It's impossible to get a significant MW result with Ns of 3 and 4. $\endgroup$ – Jeremy Miles Apr 5 '13 at 23:02
  • 1
    $\begingroup$ @JeremyMiles At least if you work at the 5% level, since the smallest attainable level is 5.7% -- but if your n's are perforce 3 and 4 respectively (rather than something you can change), one might reasonably criticize insisting on $\leq$5% in such circumstances; indeed, one might well mount an argument for a substantially higher $\alpha$, such as, oh, something more like, say about 11.4%. There's little point in keeping $\alpha$ very low if $\beta$ is really high. $\endgroup$ – Glen_b -Reinstate Monica Jan 2 '15 at 1:29
  • $\begingroup$ @Glen_b good point, I should have qualified that. (And nice solution to use a higher alpha). $\endgroup$ – Jeremy Miles Jan 2 '15 at 16:53
  • $\begingroup$ @JeremyMiles the tradeoff between the two error rates depends heavily on context of course. $\endgroup$ – Glen_b -Reinstate Monica Jan 2 '15 at 22:37
7
$\begingroup$

Does that mean they just test the same hypothesis/are useful in the same situations or are they are supposed to give the exact same p-values?

It means:

(i) the test statistics will be monotonic transformations of each other.

(ii) they give the same p-values, if you work out the p-values correctly.

Your problem is that the t-test on ranks doesn't have the distribution you use when you use t-tables (though in large samples they'll be close). You need to calculate its true distribution to correctly calculate the p-value. It matters most for small samples ... but they're also the ones where you can calculate the actual distribution most easily.

$\endgroup$
  • $\begingroup$ So correct me if I am wrong but, what you are saying is the M_W test is giving the correct p-value while the t-test on ranks is giving a value which is slightly inaccurate. OK that helps. Thanks. $\endgroup$ – Jimj Apr 6 '13 at 0:53
  • $\begingroup$ Yes, that's it exactly. Indeed, if you can see that the two statistics are monotonic versions of each other, it's easy to see that this cannot be otherwise. Showing it is not difficult algebraically. $\endgroup$ – Glen_b -Reinstate Monica Apr 6 '13 at 1:37
  • $\begingroup$ Well, it's possible that the M-W may be giving an approximation as well, depending on the circumstances, but in small samples most packages give the exact p-values. In larger samples they may switch to normal approximation (which tends to be pretty accurate unless there's a lot of ties). Where both are approximations, the M-W will generally be very close to the true p-value. $\endgroup$ – Glen_b -Reinstate Monica Apr 6 '13 at 1:37
  • $\begingroup$ And the reason a ttest on ranks is not giving the exact right pvalue is because ranked data is not normal? $\endgroup$ – Jimj Apr 7 '13 at 18:24
  • 2
    $\begingroup$ Correct. Not only is ranked data not normal, which would be sufficient to make the result not have a t-distribution, but given the sample sizes, you also know the variance under the null. $\endgroup$ – Glen_b -Reinstate Monica Apr 7 '13 at 22:51
0
$\begingroup$

N.B. Not really an answer per se...

Jonas Kristoffer Lindeløv wrote a recent blog post here that addresses relationships between linear models and group tests (such as t-test, Mann-Whitney, Wilcox, Kruskal-Wallis, ANOVA, etc.). He also created a somewhat limited simulation to assess the differences between the t-test and MW directly.

It would be nice to either have 1) an analytic calculation or 2) a strong set of simulations to create some good rules-of-thumb for when we can use a linear model on ranks as opposed to a nonparametric test, as these are sometimes more convenient.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.