I am trying to understand the limits of permutation testing.

I wrote up a small permutation test example in R, and found that permutation testing seems to fail when using absolute difference in variance as a test statistic between two samples, but works perfectly well when using standard deviation. By fail, I mean that the test commits a type-II error (p-values too high). It would be great if I could understand why this is:


Are hypothesis tests generally less powerful when using difference in variance as a test statistic, compared to difference in standard deviation? Or is difference in variance not an appropriate test statistic for permutation testing?


Here are 5 p-values produced using permutation testing for various distributions. 1000 samples are drawn from each distribution. Each permutation test then has 1000 iterations where the combined vector has 2000 samples that are randomly partitioned into two new sets of size 1000.

N(0,1) vs N(100,1) -- compare normal distributions with same variance and different means:
With var:[1] 0.992 0.991 0.992 0.994 0.990
With sd:[1] 0.641 0.624 0.627 0.609 0.650

N(0,1) vs N(100,1.5) -- compare normal distributions with different variance and different means:
Resulting p-value should be low!
With var:[1] 0.825 0.813 0.813 0.835 0.791
With sd:[1] 0 0 0 0 0

SN(0, 1, 10) vs SN(100, 1, 10) -- compare skew-normal distributions with same variance and different means:
With var:[1] 1.000 0.998 0.999 1.000 1.000
With sd:[1] 0.974 0.965 0.966 0.961 0.971

SN(0, 1, 10) vs SN(100, 1, -10) -- compare skew-normal distributions with same variance, different means, and opposite slant parameters:
With var:[1] 0.987 0.991 0.994 0.991 0.986
With sd:[1] 0.414 0.444 0.408 0.432 0.419

SN(0, 1, 10) vs SN(100, 1.5, -10) -- compare skew-normal distributions with different variance, different means, and opposite slant parameters:
Resulting p-value should be low!
With var:[1] 0.887 0.893 0.911 0.893 0.880
With sd:[1] 0 0 0 0 0


For those interested, my R code is here, and much simpler R code (not written by me) is here. Both produce the same results! My code resamples without replacement, by hashing each produced permutation and checking against each previous permutation. The simpler code does not check for repeats. In either case, the probability of a repeated resample is insignificant (there are $2000 \choose 1000$ ways to partition the combined set into two new sets).

Possibly helpful?

In Permutation, Parametric, and Bootstrap Tests of Hypotheses by Phillip Good, he provides an example on page 58, section 3.7.2, where a permutation testing approach would not work for comparing variance between samples. I don't quite follow why it doesn't work, but he says:

At first glance, the permutation test for comparing the variances of two populations would appear to be an immediate extension of the test we use for comparing location parameters, in which we use the squares of the observations rather than the observations themselves. But these squares are actually the sum of two components, one of which depends upon the unknown variance, the other upon the unknown location parameter. That is, $EX^2 = E(X − \mu + \mu)^2 = E(X − \mu)^2 + 2 \mu E(X − \mu) + \mu^2 = \sigma^2 + 0 + \mu^2.$

  • 3
    $\begingroup$ If the null is true, the p-value should be essentially uniform (not quite, because of the discreteness) - i.e. sometimes high, but not always. I have little desire to examine your code (indeed, this is the wrong group if that's what you're after), so you should make your question more statistical. What is your basis (i.e. show pictures or other summaries of results plus the reasoning that was used) for saying you get different results when using var and sd? How many runs did you do? What sample sizes? What does Good say on p58 that indicates it would not work? $\endgroup$ – Glen_b Oct 18 '14 at 23:34
  • $\begingroup$ Sorry for the lack of clarity in the original question, I've tried to make it more clear. $\endgroup$ – Sarkom Oct 19 '14 at 23:11
  • 1
    $\begingroup$ You haven't explained what you're doing, though, you're only presenting p-values. What's your test statistic in each case? [If you do something sensible, variance and standard deviation will both put the same partial order on the set of permutations and so must yield the same p-value. Are you comparing them on the same resamples?] $\endgroup$ – Glen_b Oct 20 '14 at 1:13
  • 2
    $\begingroup$ Your code undoubtedly is incorrect. $\endgroup$ – whuber Oct 20 '14 at 15:32
  • 2
    $\begingroup$ A glance at the code shows your test statistics are the difference in sample standard deviations & the difference in sample variances; these don't order the permutations the same way (ratios would). $\endgroup$ – Scortchi - Reinstate Monica Oct 21 '14 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.