Wilcoxon Rank Sum Test vs $t$-test Power Simulation

Question

It is well known that the Wilcoxon rank sum test is more powerful for detecting shifts in location when the data is non-normal. However, I am conducting a brief simulation study and my results contradict this.

One setting in my simulation strategy is as follows:

• Generate 25 random variables $X_1,\ldots,X_{25}$ such that $X\sim\mathsf{exp}(\mu=1)$
• Generate 25 random variables $Y_1,\ldots,Y_{25}$ such that $Y\sim\mathsf{exp}(\mu=1.5)$
• Determine whether the null hypothesis that $\mu_X=\mu_Y$ is rejected
• Repeat 10,000 and calculate the average

This average gives the empirical power of the test. However, I find that the $t$-test is consistently more powerful than the Wilcoxon rank sum test. For instance, in my last simulation, I obtained powers of 0.2317 and 0.2585 for the Wilcoxon and $t$-test, respectively.

Is there a flaw in my simulation strategy that is leading to unintuitive results? My R code:

power=0
for(i in 1:10000){
  x   <- rexp(25,1/1)
  y   <- rexp(25,1/1.5)
  res <- wilcox.test(x, y, alternative = "two.sided")
  power=power+(res$p.value<0.05)
}
power/10000

power=0
for(i in 1:10000){
  x   <- rexp(25,1/1)
  y   <- rexp(25,1/1.5)
  res <- t.test(x, y, alternative = "two.sided")
  power=power+(res$p.value<0.05)
}
power/10000

Answer 1

You say "It is well known that the Wilcoxon rank sum test is more powerful for detecting shifts in location when the data is non-normal" but stated so generally, this is not actually the case. It is not well known at all (for all that it might be widely believed), because it's not true.

For some non-normal distributions, sure, but not for all of them.

For example, with a beta(2,2) distribution, the relative power for the Wilcoxon vs t is even worse than it is at the normal (specifically, the asymptotic relative efficiency is about 86% vs about 95% at the normal).

Generally speaking the Wilcoxon will beat the t on power with shift alternatives on heavier-tailed symmetric distributions, but outside that it's sometimes less powerful. What makes it a good choice, however, is that it's not much less powerful (while the t can sometimes be much less powerful than the Wilcoxon).

---

I should point out that your code does not compare a location shift alternative; you have a change of scale there. A location shift could be obtained by adding something to the y-values instead.

(However, with strictly positive random variables, changes of scale may well make more sense to investigate, since those may more typically be the sort of thing you would see.)

---

However, one thing to note when comparing under such situations is that the type I error rate (actual significance level) of the t can be impacted - which will move the whole power curve up or down. To my mind it would make sense to consider separately the effect on $α$ and the effect on power at the same true significance level. This involves figuring out the effect on the significance level first and then adjusting the nominal significance level to compare power at the same true significance level, so that you're correctly interpreting the cause (e.g. is a lower rejection rate mainly due to conservatism or is it lower curvature of the power function?).

---

You can do slightly better in your simulation by comparing on the same samples. This reduces the variation. For the power comparison you were using in your question, you could do something like this:

lam1 <- 1/1.0; n1 <- 25
lam2 <- 1/1.5; n2 <- 25
nsim <- 10000
ps <- replicate(nsim,{
        x=rexp(n1,lam1)
        y=rexp(n2,lam2)
        c(wp=wilcox.test(x,y)$p.value,tp=t.test(x,y)$p.value)})
(power<-rowMeans(ps<=0.05))