Wilcoxon Rank Sum Test vs $t$-test Power Simulation

It is well known that the Wilcoxon rank sum test is more powerful for detecting shifts in location when the data is non-normal. However, I am conducting a brief simulation study and my results contradict this.

One setting in my simulation strategy is as follows:

• Generate 25 random variables $$X_1,\ldots,X_{25}$$ such that $$X\sim\mathsf{exp}(\mu=1)$$
• Generate 25 random variables $$Y_1,\ldots,Y_{25}$$ such that $$Y\sim\mathsf{exp}(\mu=1.5)$$
• Determine whether the null hypothesis that $$\mu_X=\mu_Y$$ is rejected
• Repeat 10,000 and calculate the average

This average gives the empirical power of the test. However, I find that the $$t$$-test is consistently more powerful than the Wilcoxon rank sum test. For instance, in my last simulation, I obtained powers of 0.2317 and 0.2585 for the Wilcoxon and $$t$$-test, respectively.

Is there a flaw in my simulation strategy that is leading to unintuitive results? My R code:

power=0
for(i in 1:10000){
x   <- rexp(25,1/1)
y   <- rexp(25,1/1.5)
res <- wilcox.test(x, y, alternative = "two.sided")
power=power+(res$p.value<0.05) } power/10000 power=0 for(i in 1:10000){ x <- rexp(25,1/1) y <- rexp(25,1/1.5) res <- t.test(x, y, alternative = "two.sided") power=power+(res$p.value<0.05)
}
power/10000


You say "It is well known that the Wilcoxon rank sum test is more powerful for detecting shifts in location when the data is non-normal" but stated so generally, this is not actually the case. It is not well known at all (for all that it might be widely believed), because it's not true.

For some non-normal distributions, sure, but not for all of them.

For example, with a beta(2,2) distribution, the relative power for the Wilcoxon vs t is even worse than it is at the normal (specifically, the asymptotic relative efficiency is about 86% vs about 95% at the normal).

Generally speaking the Wilcoxon will beat the t on power with shift alternatives on heavier-tailed symmetric distributions, but outside that it's sometimes less powerful. What makes it a good choice, however, is that it's not much less powerful (while the t can sometimes be much less powerful than the Wilcoxon).

I should point out that your code does not compare a location shift alternative; you have a change of scale there. A location shift could be obtained by adding something to the y-values instead.

(However, with strictly positive random variables, changes of scale may well make more sense to investigate, since those may more typically be the sort of thing you would see.)

However, one thing to note when comparing under such situations is that the type I error rate (actual significance level) of the t can be impacted - which will move the whole power curve up or down. To my mind it would make sense to consider separately the effect on $$α$$ and the effect on power at the same true significance level. This involves figuring out the effect on the significance level first and then adjusting the nominal significance level to compare power at the same true significance level, so that you're correctly interpreting the cause (e.g. is a lower rejection rate mainly due to conservatism or is it lower curvature of the power function?).

You can do slightly better in your simulation by comparing on the same samples. This reduces the variation. For the power comparison you were using in your question, you could do something like this:

lam1 <- 1/1.0; n1 <- 25
lam2 <- 1/1.5; n2 <- 25
nsim <- 10000
ps <- replicate(nsim,{
x=rexp(n1,lam1)
y=rexp(n2,lam2)
c(wp=wilcox.test(x,y)$$p.value,tp=t.test(x,y)$$p.value)})
(power<-rowMeans(ps<=0.05))

• I thought the Wilcoxon test would shine in the exponential case because the distribution is one-sided. Instead I should consider something like the Cauchy distribution? – Remy Mar 30 '20 at 8:22
• I just tried this where $X\sim N(0,1)$ and $Y\sim\mathsf{Cauchy}(0,1)$ and obtained powers of $0.0201$ and $0.0549$ for the $t$-test and Wilcoxon test, respectively. Are there any other distributions you think I should consider? – Remy Mar 30 '20 at 8:29
• You have no location shift there, so you're looking at the null (and two different distributions for X and Y, so the assumptions of both tests under the null don't hold). – Glen_b Mar 30 '20 at 8:34
• To see the Wilcoxon beat the t, try two of the same distribution, apart from a location shift. e.g. two logistic distrubutions or two sech distributions or two t-distributions with low d.f. (say $t_3$, or even Cauchy). – Glen_b Mar 30 '20 at 8:36
• If you adjust the t, trial and error is pretty much it (well, actually you can do better than naked trial and error but it would take several attempts to get it). However, if you know the significance level of the t you can do the Wilcoxon at that level. Or at least you could if the Wilcoxon itself were not discrete; it too is not operating at exactly $\alpha$ (which you should also worry about for the same reason as adjusting for the possible impact on the t). – Glen_b Mar 30 '20 at 9:09