# Selection of test statistic for permutation test of variability

I have two arrays, $$x_1$$ and $$x_2$$ and want to see if there is a statistically significant difference in variance between them. A permutation test makes sense, where the test statistic would be the absolute value of the difference in variance between each resampled group. A p-value could then be obtained by looking at the proportion of the test static results that exceed the value of the absolute difference in variance in the original samples.

My question is: I've seen in various articles online that it's "better" to use the ratio of the variances as the test statistic. My question is: how is this done specifically, and why is it better? I guess you compute the ratio of the variances between the groups on each iteration, but should you repeat the procedure and switch which group variance goes on the numerator? How would you ultimately get a p-value in this case?

And is it better because it weighs small differences in variances proportionally with the magnitude of the variance? Or am I missing something else?

• My answer shows permutation tests based on three metrics of difference in dispersion between two samples. Jun 22, 2020 at 6:58
• Thanks for your answer -- did you see my follow up? Jun 23, 2020 at 14:59

Certainly, it is traditional to compare two variances of normal samples by looking at their ratio, which is distributed according to an F-distribution. So I will use the ratio of variances as the metric for an initial permutation test.

Suppose we have samples of sizes $$n_1 = 100, n_2 =150$$ and we wish to test $$H_0: \sigma_1^2/\sigma_2^2 = 1$$ against $$H_a: \sigma_1^2/\sigma_2^2 > 1$$

Consider data sampled from normal distributions with $$\sigma_1 = 5$$ and $$\sigma_2 = 4.$$ A traditional test in R using the F-statistic and F-distribution rejects $$H_0$$ with P-value $$0.0035.$$

set.seed(612)
x1 = rnorm(100, 50, 5)
x2 = rnorm(150, 50, 4)
var.test(x1,x2, alt="g")

F test to compare two variances

data:  x1 and x2
F = 1.6296, num df = 99, denom df = 149, p-value = 0.003457
alternative hypothesis: true ratio of variances is greater than 1
95 percent confidence interval:
1.210243      Inf
sample estimates:
ratio of variances
1.629598


A permutation test with the F-ratio as metric is shown below. At each iteration the required permutation is done using the sample function on the group indexes. Not surprisingly, it rejects $$H_0$$ with about the same P-value 0.0045.

x = c(x1,x2);  g = rep(1:2, c(100,150))
f.obs = var(x[g==1])/var(x[g==2])

m = 10^5;  f.prm = numeric(m)
for(i in 1:m) {
g.prm = sample(g)
f.prm[i] = var(x[g.prm==1])/var(x[g.prm==2])  }
mean(f.prm >= f.obs)
 0.00454


A histogram of the simulated permutation distribution, along with the observed F-statistic and the density of $$\mathsf{F}(99,149),$$ is shown below. (The permutation distribution of F-statistics is about the same as the distribution used in standard tests of variances.) mh = "Simulated Permutation Dist'n of F-ratio with Density of F(99,149)"
hist(f.prm, prob=T, ylim=c(0,2.2), col="skyblue2", main=mh)
abline(v=f.obs, col="red")


An entirely different metric is the ratio of the interquartile ranges of the two samples. This metric works, but its power is not quite as good as for F-ratios. I have decreased the variance of the second population so that the revised permutation test can still find a difference (P-value=$$0.016).$$

set.seed(612)
x1 = rnorm(100, 50, 5)
x2 = rnorm(150, 50, 3.5)  # note change

x = c(x1,x2);  g = rep(1:2, c(100,150))
r.obs = IQR(x[g==1])/IQR(x[g==2]);  r.obs
 1.49377

m = 10^5;  r.prm = numeric(m)
for(i in 1:m) {
g.prm = sample(g)
r.prm[i] = IQR(x[g.prm==1])/IQR(x[g.prm==2])  }
mean(r.prm >= r.obs)
 0.01567

mh = "Simulated Permutation Dist'n of Ratio of IQRs"
hist(r.prm, prob=T, col="skyblue2", main=mh)
abline(v=r.obs, col="red", lwd=2) Finally, I have modified the R code to use the difference of standard deviations as the metric. The simulated data are the same as for the IQRs just above. The P-value is very small.

x = c(x1,x2);  g = rep(1:2, c(100,150))
d.obs = sd(x[g==1])-sd(x[g==2]);  d.obs
 1.735629

m = 10^5;  d.prm = numeric(m)
for(i in 1:m) {
g.prm = sample(g)
d.prm[i] = sd(x[g.prm==1]) - sd(x[g.prm==2])  }
mean(d.prm >= d.obs)
 4e-05

mh = "Simulated Permutation Dist'n of Difference of SDs"
hist(d.prm, prob=T, col="skyblue2", main=mh)
abline(v=d.obs, col="red", lwd=2) I hope these examples have shown that various metrics are feasible for permutations tests of homoscedasticity of two samples. Notice, even for normal data, we don't know the theoretical null distributions for ratios of IQRs or differences of standard deviations.

Note: Even though we are testing for differences is variances, there is no problem regarding the samples as exchangeable because the null hypothesis governs the permutation. More generally, however one has to make sure of exchangeability for the metric under $$H_0$$ before doing a permutation test. See this Q&A for further discussion of exchangeability.

• So if I understand your last paragraph correctly, is it accurate to say that in testing to see if $\sigma_1/\sigma_2>1$, we are testing to see if there are significant differences at all? Because, certainly $\sigma_2$ could be greater, no? Jun 22, 2020 at 13:56
• You could do a two-sided permutation test. I chose a one-sided F test to start by answer because it is easier to talk about one-sided test for ratio. // For a two-sided permutation test, it seems cleaner to look at differences. In my last example the final statement for P-val would be mean(abs(d.prm)>=abs(d.obs)). Jun 23, 2020 at 16:26
• That makes sense -- I couldn't figure out how to carry out a two-sided test with a ratio-based statistic. Thanks! Jun 23, 2020 at 20:27