# What is the non-asymptotic relative power of non-parametric tests?

The relative asymptotic power of non-parametric tests is reported to be high. What is the power of tests such as Wilcoxon Signed Rank Test and Mann-Whitney U with N's of 10, 20, 50, for example, as compared with t-tests?

-
The relative power depends on what the alternative is. With this sample size, the easiest way to determine the relative power is to run a simulation study, under alternatives you believe are plausible. This paper gives examples. –  guest May 6 '12 at 4:02
I am more used to seeing nonparametric tests compared to parametric tests via relative efficiency when the parametric test is fully efficient. It tells how much larger the sample size has to be for the nonparametric test to attain the same power as the parametric test. I would assume that relative power looks at it the other way namely for a fixed sample size how much lower is the power of the nonparametric test relative to the optimal parametric one. Power and sample size for any procedure depend on the specific alternative but relative efficiency or realtive power does not. –  Michael Chernick May 6 '12 at 14:28
Your suggestion to run a simulation makes sense, but that assumes you know the underlying distribution and also takes time, neither of which may be available. Has this type of simulation research been done for various distributions and is there a general rule of thumb to estimate the relative power of a non-parametric test under various, plausible conditions (e.g., distribution is: normal, uniform, Poisson) for various N's, say 10 to 50? –  Joel W. May 6 '12 at 15:11
Running the simulation under plausible alternatives isn't the same as knowing the truth; the efficiency changes smoothly as we move through model space, so with a good approximation of the true alternative you get a good approximation of the efficiency. It's also not more than a few lines of R code to do simulations; write commands to generate data and to compute its relevant test output, then wrap them inside replicate(). The paper I linked contains simulation research. If you want pretty math instead, this is a classic paper. –  guest May 6 '12 at 23:58
Such research would have to be vast, Joel, to cover all that ground. When you are confronting a particular problem, such as the analysis of a dataset or design of data collection, the scope is considerably narrowed. Here, @guest's suggestion is excellent: with modern software like R, Mathematica, or Matlab, you can often conduct small-sample simulations across a reasonable spectrum of possibilities in a matter of minutes, learn from the results, and move on. For simple tests, it's usually faster to just write the code than research the answer in the literature. –  whuber May 9 '12 at 16:57

As mentioned in the comments, this question has no answer because it depends on extra assumptions: the alpha level of the test, the distribution of your sample and whether or not you assume independence.

If you make those assumptions, we can get the answer with R. For example, assuming that variables are Gaussian and independent you can get the relative power with the following code:

relative_power <- function(n, delta, alpha = 0.05, resampling = 100000) {
# n: size of each Gaussian sample (with variance = 1).
# delta: difference in mean between the samples.
# alpha: level of the test (default 5%).
# resampling: number of simulations (default 10000).
reject_t <- reject_w <- rep(NA, resampling)
for (i in 1:resampling) {
# Here we assume that sampling is Gaussian and independent.
# Change those lines to whatever distribution you assume.
sample_A <- rnorm(n)
sample_B <- rnorm(n, mean = delta)

# Here we assume that we compare a t test to a Wilcoxon test.
# Change those lines to whatever tests you want to compare.
reject_t[i] <- t.test(sample_A, sample_B)$p.value < alpha reject_w[i] <- wilcox.test(sample_A, sample_B)$p.value < alpha
}
# Return the ratio of rejection of the Wilcoxon test to t test.
return (sum(reject_w) / sum(reject_t))
}


With this you can get a relative power curve as follows.

r_pow_30_5_percent <- rep(NA, 21)
delta <- seq(from = -1, to = 1, length.out = 21)
# The following loop takes a bit of time...
for (i in 1:21) {
r_pow_30_5_percent[i] <- relative_power(n = 30, delta[i])
}
plot(delta, r_pow_30_5_percent, xlab = "delta",
ylab = "Relative power", type = 'l')


To save you the trouble of running this (quite long) simulation, here is the picture I got.

As you can see, the relative power is W-shaped with a minimum for delta around 0.3 standard deviations. Yet the variation is not very large: in this particular case, the relative power does not vary by more than 5%. Also note that the value for delta = 0 should be exactly 1 because both tests have exactly the same chance of rejecting the null hypothesis (namely 5% in this case).

-
Thank you for providing this elegant simulation! However, I wonder if the results make sense. Is there so little information in the interval level data that a rank order test that ignores that information can be 96 to 99% as effective as a parametric test that uses that information (with an N of only 30)? –  Joel W. May 10 '12 at 12:48
Yes, the W shape was expected, but the fact that the relative power does not vary by more than 5% surprised me too. As far as I can judge I do not see any error in the code, so this means that the power cost of using Wilcoxon test is smaller than claimed (in this particular case). –  gui11aume May 13 '12 at 22:15
Does the relative power drop off quickly for smaller N's? What might the relative power be for an N of 5? (I have yet to learn R, unfortunately.) –  Joel W. May 14 '12 at 18:37