# Why is permutation test wrong with variance but correct with standard deviation?

I am trying to understand the limits of permutation testing.

I wrote up a small permutation test example in R, and found that permutation testing seems to fail when using absolute difference in variance as a test statistic between two samples, but works perfectly well when using standard deviation. By fail, I mean that the test commits a type-II error (p-values too high). It would be great if I could understand why this is:

## Question

Are hypothesis tests generally less powerful when using difference in variance as a test statistic, compared to difference in standard deviation? Or is difference in variance not an appropriate test statistic for permutation testing?

### Example

Here are 5 p-values produced using permutation testing for various distributions. 1000 samples are drawn from each distribution. Each permutation test then has 1000 iterations where the combined vector has 2000 samples that are randomly partitioned into two new sets of size 1000.

N(0,1) vs N(100,1) -- compare normal distributions with same variance and different means:
With var:[1] 0.992 0.991 0.992 0.994 0.990
With sd:[1] 0.641 0.624 0.627 0.609 0.650

N(0,1) vs N(100,1.5) -- compare normal distributions with different variance and different means:
Resulting p-value should be low!
With var:[1] 0.825 0.813 0.813 0.835 0.791
With sd:[1] 0 0 0 0 0

SN(0, 1, 10) vs SN(100, 1, 10) -- compare skew-normal distributions with same variance and different means:
With var:[1] 1.000 0.998 0.999 1.000 1.000
With sd:[1] 0.974 0.965 0.966 0.961 0.971

SN(0, 1, 10) vs SN(100, 1, -10) -- compare skew-normal distributions with same variance, different means, and opposite slant parameters:
With var:[1] 0.987 0.991 0.994 0.991 0.986
With sd:[1] 0.414 0.444 0.408 0.432 0.419

SN(0, 1, 10) vs SN(100, 1.5, -10) -- compare skew-normal distributions with different variance, different means, and opposite slant parameters:
Resulting p-value should be low!
With var:[1] 0.887 0.893 0.911 0.893 0.880
With sd:[1] 0 0 0 0 0


### Code

For those interested, my R code is here, and much simpler R code (not written by me) is here. Both produce the same results! My code resamples without replacement, by hashing each produced permutation and checking against each previous permutation. The simpler code does not check for repeats. In either case, the probability of a repeated resample is insignificant (there are $2000 \choose 1000$ ways to partition the combined set into two new sets).

At first glance, the permutation test for comparing the variances of two populations would appear to be an immediate extension of the test we use for comparing location parameters, in which we use the squares of the observations rather than the observations themselves. But these squares are actually the sum of two components, one of which depends upon the unknown variance, the other upon the unknown location parameter. That is, $EX^2 = E(X − \mu + \mu)^2 = E(X − \mu)^2 + 2 \mu E(X − \mu) + \mu^2 = \sigma^2 + 0 + \mu^2.$