26
$\begingroup$

This is a followup question to what Frank Harrell wrote here:

In my experience the required sample size for the t distribution to be accurate is often larger than the sample size at hand. The Wilcoxon signed-rank test is extremely efficient as you said, and it is robust, so I almost always prefer it over the t test

If I understand it correctly - when comparing the location of two unmatched samples, we would prefer to use the Wilcoxon rank-sum test over the unpaired t-test, if our sample sizes are small.

Is there a theoretical situation where we would prefer the Wilcoxon rank-sum test over the unpaired t-test, even that the sample sizes of our two groups are relatively large?

My motivation for this question stems from the observation that for a single sample t-test, using it for a not-so-small sample of a skewed distribution will yield a wrong type I error:

n1 <- 100
mean1 <- 50
R <- 100000
P_y1 <- numeric(R)
for(i in seq_len(R))
{
    y1 <- rexp(n1, 1/mean1)
    P_y1[i] <- t.test(y1 , mu = mean1)$p.value
}
sum(P_y1<.05) / R # for n1=n2=100 -> 0.0572  # "wrong" type I error
Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ To me, 0.0572 seems close enough to 0.05. $\endgroup$
    – mark999
    Dec 12, 2011 at 9:42
  • $\begingroup$ Hi Mark - when conducted under 100000 repetition of the null hypothesis, we do not expect to get this level of difference from 0.05. Generally we would expect a difference of plus minus something like two times of sqrt(0.05*0.95/100000) from 0.05 $\endgroup$
    – Tal Galili
    Dec 12, 2011 at 13:47
  • 1
    $\begingroup$ I agree that it's incorrect. I just meant that it seems close enough for practical purposes. $\endgroup$
    – mark999
    Dec 12, 2011 at 18:08
  • 1
    $\begingroup$ A related question: How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples, which considers both paired and unpaired tests, as well as alternatives to Wilcoxon such as Brunner-Munzel. There is also an excellent answer by Frank Harrell which explains why he feels justified in his approach in more detail than the above extract (e.g. the importance of the invariance of ranks under monotonic transformation). $\endgroup$
    – Silverfish
    Mar 27, 2015 at 11:47
  • $\begingroup$ @TalGalili : why would you not expect to get this level of difference, considering you are doing a t-test and the normality assumption is violated (I see that your sample is exponentially distributed)? I am asking from a novice perspective, here. I am just trying to understand what we are expecting, doing a one sample t-test when the normality assumption is violated. Why should the average type I error be smaller than 5%, or bigger than 5%, or anything? The way I see it, what we are testing is $H_0: \mu=50$ and the distribution is normal. $\endgroup$
    – Erosennin
    Apr 2, 2015 at 20:05

2 Answers 2

Reset to default
24
$\begingroup$

Yes, there is. For example, any sampling from distributions with infinite variance will wreck the t-test, but not the Wilcoxon. Referring to Nonparametric Statistical Methods (Hollander and Wolfe), I see that the asymptotic relative efficiency (ARE) of the Wilcoxon relative to the t test is 1.0 for the Uniform distribution, 1.097 (i.e., Wilcoxon is better) for the Logistic, 1.5 for the double Exponential (Laplace), and 3.0 for the Exponential.

Hodges and Lehmann showed that the minimum ARE of the Wilcoxon relative to any other test is 0.864, so you can never lose more than about 14% efficiency using it relative to anything else. (Of course, this is an asymptotic result.) Consequently, Frank Harrell's use of the Wilcoxon as a default should probably be adopted by almost everyone, including myself.

Edit: Responding to the followup question in comments, for those who prefer confidence intervals, the Hodges-Lehmann estimator is the estimator that "corresponds" to the Wilcoxon test, and confidence intervals can be constructed around that.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Is there an easy way to get a confidence interval if the Wilcoxon test is used? It seems to encourage people to put too much emphasis on the p-value, even more than they would with a parametric method. $\endgroup$
    – mark999
    Dec 12, 2011 at 9:50
  • $\begingroup$ Yes, the Hodges-Lehmann estimator is the relevant estimator, and I've edited the body of the response so that future readers don't have to go through the comments. $\endgroup$
    – jbowman
    Dec 12, 2011 at 15:09
  • 3
    $\begingroup$ biostat.mc.vanderbilt.edu/WilcoxonSoftware shows how to use R to get the Hodges-Lehmann estimate and its confidence interval. $\endgroup$
    – Frank Harrell
    Dec 12, 2011 at 23:07
  • 2
    $\begingroup$ (+1) from a stodgy anti-rank traditionalist. However, a challenge for rank tests is that the hypothesis is vague. It is generally not the same hypothesis as the t-test. The t-test tests a mean difference always, Wilcoxon tests a weighted mean rank difference. Certainly if the rank-mean difference is statistically significant, we know the distributions must differ, even if their means are the same. Neither test is powered to detect distributional differences in all cases. I only say as much because I favor interpretability. (1/2) $\endgroup$
    – AdamO
    Mar 16, 2018 at 18:45
  • 1
    $\begingroup$ A bigger challenge yet is for analysts to pre-specify such a test rather than fall back to it when the t-test is "non-significant". (2/2) $\endgroup$
    – AdamO
    Mar 16, 2018 at 18:46
25
$\begingroup$

Let me bring you back to our discussion in comments to this your question. Wilcoxon sum-rank test is equivalent to Mann-Whitney U test (and its direct extension for more-than-two samples is called Kruskal-Wallis test). You can see in Wikipedia as well as in this text that Mann-Whitney (or Kruskal-Wallis) generally compares not means or medians. It compares the overall prevalence of values: which of the samples is "stochastically greater". The test is distribution-free. T-test compares means. It assumes normal distribution. So, the tests engage in different hypotheses. In most cases, we don't plan to compare specifically the means, rather, we want to know which sample is greater by values, and it makes Mann-Whitney the default test for us. On the other hand, when both distributions are symmetric the task of testing whether one sample is "greater" than the other degenerates into the task of comparing the two means, and then, if the distributions are normal with equal variances t-test becomes somewhat more powerful.

Improve this answer
$\endgroup$
3
  • $\begingroup$ +1 for tying your answer back to the meaning of the hypotheses being tested. $\endgroup$
    – Josh Hemann
    Dec 12, 2011 at 16:51
  • 1
    $\begingroup$ By "which of the samples is "stochastically greater" ", do you mean "which one of the samples generally take greater values compared to the other"? If not, what do u mean? Could u elaborate on this a little further please? $\endgroup$
    – Erdogan CEVHER
    Aug 1, 2018 at 8:21
  • 1
    $\begingroup$ @Erdogan, yes, we may say like you said. The strict wording is this: in a randomly chosen pair of objects, one from each sample, the object from the "stochastically more dominant" sample will be higher (by the value) than the object from the other sample with probability >0.5. $\endgroup$
    – ttnphns
    Aug 1, 2018 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.