I am trying to understand the limits of permutation testing.
I wrote up a small permutation test example in R, and found that permutation testing seems to fail when using absolute difference in variance as a test statistic between two samples, but works perfectly well when using standard deviation. By fail, I mean that the test commits a type-II error (p-values too high). It would be great if I could understand why this is:
Are hypothesis tests generally less powerful when using difference in variance as a test statistic, compared to difference in standard deviation? Or is difference in variance not an appropriate test statistic for permutation testing?
Here are 5 p-values produced using permutation testing for various distributions. 1000 samples are drawn from each distribution. Each permutation test then has 1000 iterations where the combined vector has 2000 samples that are randomly partitioned into two new sets of size 1000.
N(0,1) vs N(100,1) -- compare normal distributions with same variance and different means: With var: 0.992 0.991 0.992 0.994 0.990 With sd: 0.641 0.624 0.627 0.609 0.650 N(0,1) vs N(100,1.5) -- compare normal distributions with different variance and different means: Resulting p-value should be low! With var: 0.825 0.813 0.813 0.835 0.791 With sd: 0 0 0 0 0 SN(0, 1, 10) vs SN(100, 1, 10) -- compare skew-normal distributions with same variance and different means: With var: 1.000 0.998 0.999 1.000 1.000 With sd: 0.974 0.965 0.966 0.961 0.971 SN(0, 1, 10) vs SN(100, 1, -10) -- compare skew-normal distributions with same variance, different means, and opposite slant parameters: With var: 0.987 0.991 0.994 0.991 0.986 With sd: 0.414 0.444 0.408 0.432 0.419 SN(0, 1, 10) vs SN(100, 1.5, -10) -- compare skew-normal distributions with different variance, different means, and opposite slant parameters: Resulting p-value should be low! With var: 0.887 0.893 0.911 0.893 0.880 With sd: 0 0 0 0 0
For those interested, my R code is here, and much simpler R code (not written by me) is here. Both produce the same results! My code resamples without replacement, by hashing each produced permutation and checking against each previous permutation. The simpler code does not check for repeats. In either case, the probability of a repeated resample is insignificant (there are $2000 \choose 1000$ ways to partition the combined set into two new sets).
In Permutation, Parametric, and Bootstrap Tests of Hypotheses by Phillip Good, he provides an example on page 58, section 3.7.2, where a permutation testing approach would not work for comparing variance between samples. I don't quite follow why it doesn't work, but he says:
At first glance, the permutation test for comparing the variances of two populations would appear to be an immediate extension of the test we use for comparing location parameters, in which we use the squares of the observations rather than the observations themselves. But these squares are actually the sum of two components, one of which depends upon the unknown variance, the other upon the unknown location parameter. That is, $EX^2 = E(X − \mu + \mu)^2 = E(X − \mu)^2 + 2 \mu E(X − \mu) + \mu^2 = \sigma^2 + 0 + \mu^2.$