9
$\begingroup$

What does it mean when people say that a t-test performed on ranked data is equivalent to a Mann-Whitney U-test? Does that mean they just test the same hypothesis/are useful in the same situations or are they are supposed to give the exact same p-values? The reason I ask is I tried both in R and compared two groups with very small sample sizes (3 and 4). I got completely different answers: one significant and one not.

The two groups are A=(1,2,3) and B=(4,5,6,7).

t-test: p = 0.01

Mann-Whitney U-test: p = 0.06

Improve this question
$\endgroup$
4
  • $\begingroup$ It's impossible to get a significant MW result with Ns of 3 and 4. $\endgroup$
    – Jeremy Miles
    Apr 5, 2013 at 23:02
  • 1
    $\begingroup$ @JeremyMiles At least if you work at the 5% level, since the smallest attainable level is 5.7% -- but if your n's are perforce 3 and 4 respectively (rather than something you can change), one might reasonably criticize insisting on $\leq$5% in such circumstances; indeed, one might well mount an argument for a substantially higher $\alpha$, such as, oh, something more like, say about 11.4%. There's little point in keeping $\alpha$ very low if $\beta$ is really high. $\endgroup$
    – Glen_b
    Jan 2, 2015 at 1:29
  • $\begingroup$ @Glen_b good point, I should have qualified that. (And nice solution to use a higher alpha). $\endgroup$
    – Jeremy Miles
    Jan 2, 2015 at 16:53
  • $\begingroup$ @JeremyMiles the tradeoff between the two error rates depends heavily on context of course. $\endgroup$
    – Glen_b
    Jan 2, 2015 at 22:37

2 Answers 2

Reset to default
7
$\begingroup$

Does that mean they just test the same hypothesis/are useful in the same situations or are they are supposed to give the exact same p-values?

It means:

(i) the test statistics will be monotonic transformations of each other.

(ii) they give the same p-values, if you work out the p-values correctly.

Your problem is that the t-test on ranks doesn't have the distribution you use when you use t-tables (though in large samples they'll be close). You need to calculate its true distribution to correctly calculate the p-value. It matters most for small samples ... but they're also the ones where you can calculate the actual distribution most easily.

Improve this answer
$\endgroup$
9
  • $\begingroup$ So correct me if I am wrong but, what you are saying is the M_W test is giving the correct p-value while the t-test on ranks is giving a value which is slightly inaccurate. OK that helps. Thanks. $\endgroup$
    – Jimj
    Apr 6, 2013 at 0:53
  • $\begingroup$ Yes, that's it exactly. Indeed, if you can see that the two statistics are monotonic versions of each other, it's easy to see that this cannot be otherwise. Showing it is not difficult algebraically. $\endgroup$
    – Glen_b
    Apr 6, 2013 at 1:37
  • $\begingroup$ Well, it's possible that the M-W may be giving an approximation as well, depending on the circumstances, but in small samples most packages give the exact p-values. In larger samples they may switch to normal approximation (which tends to be pretty accurate unless there's a lot of ties). Where both are approximations, the M-W will generally be very close to the true p-value. $\endgroup$
    – Glen_b
    Apr 6, 2013 at 1:37
  • $\begingroup$ And the reason a ttest on ranks is not giving the exact right pvalue is because ranked data is not normal? $\endgroup$
    – Jimj
    Apr 7, 2013 at 18:24
  • 2
    $\begingroup$ Correct. Not only is ranked data not normal, which would be sufficient to make the result not have a t-distribution, but given the sample sizes, you also know the variance under the null. $\endgroup$
    – Glen_b
    Apr 7, 2013 at 22:51
0
$\begingroup$

N.B. Not really an answer per se...

Jonas Kristoffer Lindeløv wrote a recent blog post here that addresses relationships between linear models and group tests (such as t-test, Mann-Whitney, Wilcox, Kruskal-Wallis, ANOVA, etc.). He also created a somewhat limited simulation to assess the differences between the t-test and MW directly.

It would be nice to either have 1) an analytic calculation or 2) a strong set of simulations to create some good rules-of-thumb for when we can use a linear model on ranks as opposed to a nonparametric test, as these are sometimes more convenient.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.