Certainly, it is traditional to compare two variances of normal samples by
looking at their ratio, which is distributed according to an F-distribution.
So I will use the ratio of variances as the metric for an initial permutation
test.
Suppose we have samples of sizes $n_1 = 100, n_2 =150$ and we wish to
test $H_0: \sigma_1^2/\sigma_2^2 = 1$ against $H_a: \sigma_1^2/\sigma_2^2 > 1$
Consider data sampled from normal distributions with $\sigma_1 = 5$ and
$\sigma_2 = 4.$ A traditional test in R using the F-statistic and F-distribution rejects $H_0$ with P-value $0.0035.$
set.seed(612)
x1 = rnorm(100, 50, 5)
x2 = rnorm(150, 50, 4)
var.test(x1,x2, alt="g")
F test to compare two variances
data: x1 and x2
F = 1.6296, num df = 99, denom df = 149, p-value = 0.003457
alternative hypothesis: true ratio of variances is greater than 1
95 percent confidence interval:
1.210243 Inf
sample estimates:
ratio of variances
1.629598
A permutation test with the F-ratio as metric is shown below.
At each iteration the required permutation is done using the sample
function on the group indexes. Not surprisingly, it rejects $H_0$
with about the same P-value 0.0045.
x = c(x1,x2); g = rep(1:2, c(100,150))
f.obs = var(x[g==1])/var(x[g==2])
m = 10^5; f.prm = numeric(m)
for(i in 1:m) {
g.prm = sample(g)
f.prm[i] = var(x[g.prm==1])/var(x[g.prm==2]) }
mean(f.prm >= f.obs)
[1] 0.00454
A histogram of the simulated permutation distribution, along with the
observed F-statistic and the density of $\mathsf{F}(99,149),$ is shown below. (The permutation distribution of F-statistics is about the same as the distribution used in standard tests of variances.)
mh = "Simulated Permutation Dist'n of F-ratio with Density of F(99,149)"
hist(f.prm, prob=T, ylim=c(0,2.2), col="skyblue2", main=mh)
abline(v=f.obs, col="red")
curve(df(x,99,149), add=T)
An entirely different metric is the ratio of the interquartile
ranges of the two samples. This metric works, but its power is not quite
as good as for F-ratios. I have decreased the variance of the second population so that
the revised permutation test can still find a difference (P-value=$0.016).$
set.seed(612)
x1 = rnorm(100, 50, 5)
x2 = rnorm(150, 50, 3.5) # note change
x = c(x1,x2); g = rep(1:2, c(100,150))
r.obs = IQR(x[g==1])/IQR(x[g==2]); r.obs
[1] 1.49377
m = 10^5; r.prm = numeric(m)
for(i in 1:m) {
g.prm = sample(g)
r.prm[i] = IQR(x[g.prm==1])/IQR(x[g.prm==2]) }
mean(r.prm >= r.obs)
[1] 0.01567
mh = "Simulated Permutation Dist'n of Ratio of IQRs"
hist(r.prm, prob=T, col="skyblue2", main=mh)
abline(v=r.obs, col="red", lwd=2)
Finally, I have modified the R code to use the difference of
standard deviations as the metric. The simulated data are the
same as for the IQRs just above. The P-value is very small.
x = c(x1,x2); g = rep(1:2, c(100,150))
d.obs = sd(x[g==1])-sd(x[g==2]); d.obs
[1] 1.735629
m = 10^5; d.prm = numeric(m)
for(i in 1:m) {
g.prm = sample(g)
d.prm[i] = sd(x[g.prm==1]) - sd(x[g.prm==2]) }
mean(d.prm >= d.obs)
[1] 4e-05
mh = "Simulated Permutation Dist'n of Difference of SDs"
hist(d.prm, prob=T, col="skyblue2", main=mh)
abline(v=d.obs, col="red", lwd=2)
I hope these examples have shown that various metrics are feasible for permutations tests of homoscedasticity of two samples. Notice, even for normal data, we don't know the theoretical null distributions for ratios of IQRs or differences of standard deviations.
Note: Even though we are testing for differences is variances,
there is no problem regarding the samples as exchangeable because
the null hypothesis governs the permutation. More generally, however
one has to make sure of exchangeability for the metric under $H_0$ before
doing a permutation test. See this Q&A for further discussion of exchangeability.
